0
votes

I'm trying to scroll all the way down inside an HTML table, it isn't the "main" scroll of the webpage, but just the scroll of the HTML table. I plan to just scroll all the way down and then find the certain html tags I want. This is my script at the moment, but the scrolling would just work if I wanted to scroll on the main document.

Essentially I'm trying to retrieve all the data from the main table after scrolling, but there is an additional issue as the table only loads X amount of rows given the dimensions of the window browser of the table, I want to able to append all of them together.

url = "https://governmentofbc.maps.arcgis.com/home/item.html?id=b8a2b437ccc24f04b975f76df6814cb1#data"
driver = webdriver.Chrome()
driver.get(url)
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(3)
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

soup = BeautifulSoup(driver.page_source, "html.parser")

1
Do you exclusively have to use selenium? From what I see you can easily get what you need using the requests library, no more worrying with scrolling. - West
yep, I need to scrape the webpage - kjay

1 Answers

1
votes

Had the same problem, therefore i can tell you what worked for me:

element = driver.find_element_by_xpath("the body you want to scroll in")
for i in range(60):
driver.execute_script("arguments[0].scrollBy(0, 500)", element)
time.sleep(2)

you can adjust those "60" depending on how far you need to scroll and how big the site is. you can also replace the time.sleep() by driver.implicitly_wait(2)