Unable to locate element, can't scrape 'reviews'

Question

I'm scraping the product reviews from the sephora website which contains javascript(reviews), but I can't scrape. This is my code:

    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support.expected_conditions import presence_of_element_located as EC
    import time
    chrome_path = '/media/danish-khan/New Volume/Web_scraping/rgcrawler2/chromedriver'
    driver = webdriver.Chrome(chrome_path)
    
    chrome_options = Options()
    url = 'https://www.sephora.com/product/the-porefessional-face-primer-P264900?skuId=1259068&icid2=products%20grid:p264900:product'
    
    driver.get(url)
    WebDriverWait(driver, 70)
    time.sleep(70)
    review = driver.find_element_by_class_name('css-1jg2pb9 eanm77i0')
    for post in review:
    #try:
    #    element = WebDriverWait(driver, 50).until(
    #        EC.presence_of_element_located((By.XPATH, "//div[@class = 'css-1jg2pb9 eanm77i0']"))
    #    )
    #finally:
    #    driver.quit()
    #
    
       print(review)
    
    
    driver.close()'

The output is:

Traceback (most recent call last): File "resgt.py", line 15, in review = driver.find_element_by_class_name('css-1jg2pb9 eanm77i0') File "/home/danish-khan/miniconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 564, in find_element_by_class_name return self.find_element(by=By.CLASS_NAME, value=name) File "/home/danish-khan/miniconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element 'value': value})['value'] File "/home/danish-khan/miniconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute self.error_handler.check_response(response) File "/home/danish-khan/miniconda3/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".css-1jg2pb9 eanm77i0"} (Session info: chrome=85.0.4183.102)

You are first looking for your CSS class and later you are waiting for it to be present. Try inverting the order and see if it works. — Maximilian Peters

Norton409 Norton409 · Accepted Answer · 2020-11-23T05:29:44

The reviews for that page are being loaded in asynchronously specifically when the section is scrolled into view. you will have to scroll to an element close to where the reviews are and wait until it appears. Only then you will be able to retrieve the element.
I was able to do this with this code

driver.execute_script("window.scrollTo(0, document.body.scrollHeight/2);")
time.sleep(10)
review = driver.find_element_by_css_selector('.css-1jg2pb9.eanm77i0')
# review = driver.find_element_by_xpath('/html/body/div[1]/div[2]/div/main/div/div[2]/div[1]/div/div[5]/div/div[2]/div[1]/div[2]')
print(review)

I left the Xpath in there as thats what i used to get it the first time note* you may have to adjust the timing and the scroll height to get it always correct

Unable to locate element, can't scrape 'reviews'

1 Answers