Read a page after another from a drop down menu - Can't find the drop down on 2nd page

Question

I've got a page.

And I want to go on every page (in order to get the URL) associated with an element of the drop down menu from the top of the page.

New to selenium, I'm trying some preliminary work:

Open the driver
Get it to webpage
Select the drop down menu
Just select a random "name" from a arbitrary value = 2
Get on the page and get the URL from it. Print it.
Just select a random "name" from a arbitrary value = 3 ERROR.

The code I use:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
import time

driver = webdriver.Firefox()
driver.get("http://www.hillsproducts.com/General.aspx/en-GB/PD/a-d-canine/original/can")
select = Select(driver.find_element_by_xpath("//select[@id='productSpecifier_product']"))
value="2"
select.select_by_value(value)
print(driver.current_url)
time.sleep(10)
value="3"
select.select_by_value(value)
print(driver.current_url)

There is something i don't get. The error i've got is the following :

Traceback (most recent call last): File "/Users/Luigi/Desktop/selenium_attempt.py", line 19, in select.select_by_value(value) File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium-2.46.1-py3.4.egg/selenium/webdriver/support/select.py", line 76, in select_by_value opts = self._el.find_elements(By.CSS_SELECTOR, css) File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium-2.46.1-py3.4.egg/selenium/webdriver/remote/webelement.py", line 485, in find_elements {"using": by, "value": value})['value'] File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium-2.46.1-py3.4.egg/selenium/webdriver/remote/webelement.py", line 447, in _execute return self._parent.execute(command, params) File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium-2.46.1-py3.4.egg/selenium/webdriver/remote/webdriver.py", line 193, in execute self.error_handler.check_response(response) File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium-2.46.1-py3.4.egg/selenium/webdriver/remote/errorhandler.py", line 181, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.StaleElementReferenceException: Message: Element not found in the cache - perhaps the page has changed since it was looked up Stacktrace: at fxdriver.cache.getElementAt (resource://fxdriver/modules/web-element-cache.js:9348) at Utils.getElementAt (file:///var/folders/8s/hl6bx6z91yq6r81hpqg995rw0000gn/T/tmpr37ozu9l/extensions/[email protected]/components/driver-component.js:8942) at FirefoxDriver.prototype.findElementsInternal_ (file:///var/folders/8s/hl6bx6z91yq6r81hpqg995rw0000gn/T/tmpr37ozu9l/extensions/[email protected]/components/driver-component.js:10685) at FirefoxDriver.prototype.findChildElements (file:///var/folders/8s/hl6bx6z91yq6r81hpqg995rw0000gn/T/tmpr37ozu9l/extensions/[email protected]/components/driver-component.js:10706) at DelayedCommand.prototype.executeInternal_/h (file:///var/folders/8s/hl6bx6z91yq6r81hpqg995rw0000gn/T/tmpr37ozu9l/extensions/[email protected]/components/command-processor.js:12643) at DelayedCommand.prototype.executeInternal_ (file:///var/folders/8s/hl6bx6z91yq6r81hpqg995rw0000gn/T/tmpr37ozu9l/extensions/[email protected]/components/command-processor.js:12648) at DelayedCommand.prototype.execute/< (file:///var/folders/8s/hl6bx6z91yq6r81hpqg995rw0000gn/T/tmpr37ozu9l/extensions/[email protected]/components/command-processor.js:12590)

Any idea would be appreciated !

UPDATE after Alex's answer :

Traceback (most recent call last): File "/Users/Luigi/Desktop/selenium_attempt.py", line 18, in if index >= len(select.options): File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium-2.46.1-py3.4.egg/selenium/webdriver/support/select.py", line 46, in options return self._el.find_elements(By.TAG_NAME, 'option') File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium-2.46.1-py3.4.egg/selenium/webdriver/remote/webelement.py", line 485, in find_elements {"using": by, "value": value})['value'] File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium-2.46.1-py3.4.egg/selenium/webdriver/remote/webelement.py", line 447, in _execute return self._parent.execute(command, params) File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium-2.46.1-py3.4.egg/selenium/webdriver/remote/webdriver.py", line 193, in execute self.error_handler.check_response(response) File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium-2.46.1-py3.4.egg/selenium/webdriver/remote/errorhandler.py", line 181, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.StaleElementReferenceException: Message: Element not found in the cache - perhaps the page has changed since it was looked up Stacktrace: at fxdriver.cache.getElementAt (resource://fxdriver/modules/web-element-cache.js:9348) at Utils.getElementAt (file:///var/folders/8s/hl6bx6z91yq6r81hpqg995rw0000gn/T/tmpzrilw39c/extensions/[email protected]/components/driver-component.js:8942) at FirefoxDriver.prototype.findElementsInternal_ (file:///var/folders/8s/hl6bx6z91yq6r81hpqg995rw0000gn/T/tmpzrilw39c/extensions/[email protected]/components/driver-component.js:10685) at FirefoxDriver.prototype.findChildElements (file:///var/folders/8s/hl6bx6z91yq6r81hpqg995rw0000gn/T/tmpzrilw39c/extensions/[email protected]/components/driver-component.js:10706) at DelayedCommand.prototype.executeInternal_/h (file:///var/folders/8s/hl6bx6z91yq6r81hpqg995rw0000gn/T/tmpzrilw39c/extensions/[email protected]/components/command-processor.js:12643) at DelayedCommand.prototype.executeInternal_ (file:///var/folders/8s/hl6bx6z91yq6r81hpqg995rw0000gn/T/tmpzrilw39c/extensions/[email protected]/components/command-processor.js:12648) at DelayedCommand.prototype.execute/< (file:///var/folders/8s/hl6bx6z91yq6r81hpqg995rw0000gn/T/tmpzrilw39c/extensions/[email protected]/components/command-processor.js:12590)

alecxe alecxe · Accepted Answer · 2015-07-28T14:00:21

You have to reinstantiate the Select() every time a new page is loaded:

from selenium import webdriver
from selenium.webdriver.support.ui import Select


driver = webdriver.Firefox()
driver.get("http://www.hillsproducts.com/General.aspx/en-GB/PD/a-d-canine/original/can")

index = 0
while True:
    select = Select(driver.find_element_by_id("productSpecifier_product"))

    # exit the loop if all the options were seen
    if index >= len(select.options):
        break

    select.select_by_index(index)
    print(driver.current_url)

    index += 1

Read a page after another from a drop down menu - Can't find the drop down on 2nd page

1 Answers