Iterating Through driver.get link Selenium Python

Question

I'm having trouble iterating through the driver.get url to scrape information from subsequent pages.

Background:

My goal is to grab the stock ticker name of all stocks on a given page, and then go to the next page and do it again.

So far, I've been able to grab the ticker name info as a list on any given page (see code and image below).

I've also been able to change the url from page 1 all the way up to page 5 (as noted by the print statement in the code below).

Where I am getting stuck is trying to iterate through the driver.get url.

I've tried using requests as well, but I get back an empty list

url = https://swingtradebot.com/equities?min_vol=250000&min_price=3&max_price=999999.0&adx_trend=&grade=&include_etfs=0&html_button=as_html&page="
stock_tickers = []

for page in range(1,10):
    page_url = url+str(page)
    
    res = requests.get(page_url)
    
    soup = bs4.BeautifulSoup(res.text,"lxml")
    
    for name in soup.find_all('a'):
        name.get('href')
        stock_tickers.append(name.text)

Is there a way to iterate through the driver.get url?

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import bs4

PATH = "C:\\Users\\MYUSERNAME\\Webdrivers\\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://swingtradebot.com/equities?min_vol=250000&min_price=3&max_price=999999.0&adx_trend=&grade=&include_etfs=0&html_button=as_html&page=1")

string = "https://swingtradebot.com/equities?min_vol=250000&min_price=3&max_price=999999.0&adx_trend=&grade=&include_etfs=0&html_button=as_html&page=1]"
new_character = '1'
position = -2
new_character_int = int(new_character)
new_string = string[:position] + str(new_character_int) + string[position+1:]

while new_character_int <=5:
    
    new_string = string[:position] + str(new_character_int) + string[position+1:]
    
    html = driver.page_source 

    soup = bs4.BeautifulSoup(html, 'html.parser')
    
    stock_tickers = []

    for name in soup.find_all('a'):
        name.get('href')
        stock_tickers.append(name.text)
        
    
    new_character_int +=1
    print(new_string)
    
stock_tickers[27:67:2]

Output from code above

Assad Ali Assad Ali · Accepted Answer · 2020-11-30T01:30:49

If you are trying to loop and your new link is new_string you must get that link in driver to be able to process the html.

Currently your print your links but you also need to get those links, see fix below

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import bs4

PATH = "C:\\Users\\MYUSERNAME\\Webdrivers\\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://swingtradebot.com/equities?min_vol=250000&min_price=3&max_price=999999.0&adx_trend=&grade=&include_etfs=0&html_button=as_html&page=1")

string = "https://swingtradebot.com/equities?min_vol=250000&min_price=3&max_price=999999.0&adx_trend=&grade=&include_etfs=0&html_button=as_html&page=1]"
new_character = '1'
position = -2
new_character_int = int(new_character)
new_string = string[:position] + str(new_character_int) + string[position+1:]

while new_character_int <=5:
    
    new_string = string[:position] + str(new_character_int) + string[position+1:]
    driver.get(new_string)
    html = driver.page_source 

    soup = bs4.BeautifulSoup(html, 'html.parser')
    
    stock_tickers = []

    for name in soup.find_all('a'):
        name.get('href')
        stock_tickers.append(name.text)
        
    
    new_character_int +=1
    print(new_string)
    
stock_tickers[27:67:2]

Iterating Through driver.get link Selenium Python

1 Answers