0
votes

I'm trying to extract real estate listing info from a site using selenium and beautiful soup using this tutorial: https://medium.com/@ben.sturm/scraping-house-listing-data-using-selenium-and-beautiful-soup-1cbb94ba9492

Aim is to gather all the href links from the first page before finding the 'next page' button, navigating to next and collecting all links on that page and so on.

Tried with a single function to achieve this and repeat for each page but can't figure out why it's not working. New to learning code and have seems too trivial to find an answer yet. Would appreciate any help

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time
import sys
import numpy as np
import pandas as pd
import regex as re


driver = webdriver.Chrome
url = "http://property.shw.co.uk/searchproperties/Level2-0/Level1-0-181-236-167-165/Units/Development-or-House-and-Flat-or-Investment-or-Land-or-Office-or-Other/UnitIds-0/For-Sale"
driver.get(url)
try:
    wait = WebDriverWait(driver, 3)
    wait.until(EC.presence_of_element_located((By.ID, "body1")))
    print("Page is Ready!")
except TimeoutException:
    print("page took too long to load")


def get_house_links(url, driver, pages=3):
    house_links = []
    driver.get(url)
    for i in range(pages):
        soup = BeautifulSoup(driver.page_source, 'html.parser')
        listings = soup.find_all("a", class_="L")
        page_data = [row['href'] for row in listings]
        house_links.append(page_data)
        time.sleep(np.random.lognormal(0, 1))
        next_button = soup.find_all("a", class_="pageingBlock darkBorder")
        next_button_link = ['http://property.shw.co.uk'+row['href'] for row in next_button]
        if i < 3:
            driver.get(next_button_link[0])
    return house_links
get_house_links(url, driver)
1
You need to explain not working, but for starters you don't call get_house_links() anywhere. - Guy
Thanks Guy, as in the browser does not move to second page and so on. I added: get_house_links(url, driver) and now it goes to the second page but not to the third and final page. Appreciate your help - Alistair Khandy
Currently it's going to page 1, refreshes that page, then to page 2, then back to 1 and then back to 2 and ends. - Alistair Khandy

1 Answers

0
votes

class_="pageingBlock darkBorder" match the previous page button as well, so next_button_link[0] send you back to previous page. You need more precise locator

next_button = soup.select('img[src*="propNext"]')
if next_button:
    next_button = next_button[0].find_parent('a')
    next_button_link = 'http://property.shw.co.uk' + next_button['href']
    driver.get(next_button_link)