Navigating to next page of a javascript table using python selenium chrome webdriver

Question

My goal is to loop through each of the pages and scrape the contents into a DataFrame. So far, I can get the first 20 no problem. I cannot seem to figure out how to navigate to the next page on a javascript table using python selenium chrome webdriver. Have tried a few of the solutions (below seems like the closest) but cannot replicate the result. I would post the website but it is one where you need log in credentials to access. Provided a page source screenshot for the relevant ul and li elements

stackoverflow solution tried: How to click on the list of the <li> elements in an <ul> elements with selenium in python?

Am able to get the first 20 rows using: driver.find_element_by_xpath('//*[@id="allPicksId"]/div/div2').text

Then, when trying to get the next rows on next table page using the code below I get an exception: next_table_page = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".ant-pagination-next"))).click()

Yields this exception: ElementClickInterceptedException: Message: element click intercepted: Element is not clickable at point (546, 996)

Any guidance would be appreciated!

Page Source Photo

Table Page Navigation Bar Photo

Answer 1

you can use for loop to handle this

example:

for I in range(1, 23):
      driver.find_element_by_xpath().click()

Or

you can use the site URL to go next page

see this url

and second url

now you must change the number of pages to go next page

Answer 2

I think I figured it out by using the code below:

element = driver.find_element_by_class_name('ant-pagination-next') driver.execute_script("arguments[0].click();", element)