Iterating with selenium through pages

Question

This is the first webpage that I've scraped, and some of the other solutions I've found don't quite seem to help. As you'll see, the "Next" button is still visible, but the CSS changes just a bit, when you get to the last page.

A few notes. I'm using python, selenium and google chrome.

I am trying to loop through each part of the table on this page: https://caearlyvoting.sos.ca.gov/

I have figured out how to loop through each county, and grab the information I need(i think). However, I am getting hung up on how to move to the next page when the table has more records than the 10 displayed by default.

I've tried variations of this

  try:
        next_page = driver.find_element_by_class_name('paginate_button')
        next_page.click()
    except NoSuchElementException:
        pass

But no luck. I've tried getting the element in different ways but I run into the same issues.

Can someone help me figure out how to click through each page, grab what I need and then move onto the next county? I don't need help grabbing the info from the table, just clicking through the pages and then moving onto the next county.

EDIT Here's the rest of the code based on a follow up. I am having difficulty structuring it.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
import pandas as pd
import time # not for production

# Name of the counties Single column with county names
county_df = pd.read_csv('Counties.csv')

# Path to driver on this computer
chrome_driver_path = r'C:\Windows\chromedriver'

# url to scrape
url = 'https://caearlyvoting.sos.ca.gov/'

with webdriver.Chrome(executable_path=chrome_driver_path)as driver:
    # Open window, maximize and set an implicit wait
    driver.get(url)
    driver.maximize_window()
    driver.implicitly_wait(10)
    actions = ActionChains(driver) #* New line here from stackoverflow
    # find the county selection
    county_selector = driver.find_element_by_id('CountyID')
    # for loop tomove through the counties
    for county in county_df['County'][:5]:
        # Input the county namne
        county_selector.send_keys(county)
        ### Code to grab data goes here
        
        ########* Code from stackoverflow ########
        while True:
            next_page = driver.find_element_by_css_selector(".paginate_button.next")
            next_bnt_classes = next_page.get_attribute("class")
            if "disabled" in next_bnt_classes:
                break  #last page reached, no more next pages, break the loop
            else:
                actions.move_to_element(next_page).perform()
                time.sleep(0.5)
                #get the actual next page button and click it
                driver.find_element_by_css_selector(".paginate_button.next a").click()

Prophet Prophet · Accepted Answer · 2021-08-19T21:36:47

You are using wrong locator.
Also the next page button can appear out of the view, on the bottom of the page, so you will have to scroll to that element and only after that click it.
On the last page the next page button is disabled.
In this case it contains disabled class name.
So your code can be:

from selenium.webdriver.common.action_chains import ActionChains

actions = ActionChains(driver)

while True:
    #grab the data from current page, after that:
    next_page = driver.find_element_by_css_selector(".paginate_button.next")
    next_bnt_classes = next_page.get_attribute("class")
    if "disabled" in next_bnt_classes:
        break  #last page reached, no more next pages, break the loop
    else:
        next_page = driver.find_element_by_css_selector(".paginate_button.next")
        actions.move_to_element(next_page).perform()
        time.sleep(0.5)
        #get the actual next page button and click it
        driver.find_element_by_css_selector(".paginate_button.next a").click()

UPD
The working code is slightly different:

from selenium.webdriver.common.action_chains import ActionChains

actions = ActionChains(driver)

while True:
    #grab the data from current page, after that:
    next_page = driver.find_element_by_css_selector(".paginate_button.next")
    next_bnt_classes = next_page.get_attribute("class")
    if next_bnt_classes == 'paginate_button next disabled':
        break  #last page reached, no more next pages, break the loop
    else:
        # Move to the next page for the county and append the data              
        next_page.click()

Iterating with selenium through pages

1 Answers