Selenium cannot get iframe

Question

I am trying to retrieve the Job Description and Job Requirements from this link: https://www.jobsbank.gov.sg/ICMSPortal/portlets/JobBankHandler/SearchDetail.do?id=JOB-2016-0008238

using selenium. This is my code:

driver = webdriver.Firefox()
url = "https://www.jobsbank.gov.sg/ICMSPortal/portlets/JobBankHandler/SearchDetail.do?id=JOB-2016-0008238"
driver.get(url)
iframes = driver.find_elements_by_xpath('//*[@id="divMainJobDescription"]')
print len(iframes)
driver.switch_to_frame(iframes)
print driver.page_source

I get the following error:

Traceback (most recent call last): File "D:\scraper.py", line 79, in main() File "D:\Abigail MTI Projects\Singapore Skills\JobsBankScraper\jobsbank14012016_abi.py", line 74, in main driver.switch_to_frame(iframes) File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 537, in switch_to_frame self._switch_to.frame(frame_reference) File "C:\Python27\lib\site-packages\selenium\webdriver\remote\switch_to.py", line 67, in frame self._driver.execute(Command.SWITCH_TO_FRAME, {'id': frame_reference}) File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 201, in execute self.error_handler.check_response(response) File "C:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 181, in check_response raise exception_class(message, screen, stacktrace) NoSuchFrameException: Message: Unable to locate frame: [object Object] Stacktrace: at FirefoxDriver.prototype.switchToFrame (file:///c:/users/mti_wl~1/appdata/local/temp/tmpwelh3x/extensions/[email protected]/components/driver-component.js:10745) at DelayedCommand.prototype.executeInternal_/h (file:///c:/users/mti_wl~1/appdata/local/temp/tmpwelh3x/extensions/[email protected]/components/command-processor.js:12551) at DelayedCommand.prototype.executeInternal_ (file:///c:/users/mti_wl~1/appdata/local/temp/tmpwelh3x/extensions/[email protected]/components/command-processor.js:12556) at DelayedCommand.prototype.execute/< (file:///c:/users/mti_wl~1/appdata/local/temp/tmpwelh3x/extensions/[email protected]/components/command-processor.js:12498)

Answer 1

iframes is list, you need to switch to a single frame.

driver.find_elements return list while driver.find_element return single web element (note the 's' in element/s).

In addition, divMainJobDescription is not iframe, I believe you are looking for frameJobDescription

iframe = driver.find_element_by_id("frameJobDescription")

Answer 2

You need to target the iframe, switch to that frame, and then you can start identifying other elements, like divs, which exist inside that frame. You need to specify the iframe tag itself and not an element inside the iframe.

So, instead of targeting a div immediately inside the frame like

iframes = driver.find_elements_by_xpath('//*[@id="divMainJobDescription"]')

target the iframe tag, then switch. (You can use the singular find_element_by_xpath if you want only one iframe.)

iframe = driver.find_element_by_xpath('//iframe[@id="frameJobDescription"]')
driver.switch_to_frame(iframe)
job_description = driver.find_element_by_xpath('//*[@id="divMainJobDescription"]')

Bonus: You can use other methods to target the iframe, such as finding the element by ID instead of writing the xpath, if you are only looking at IDs.

iframe = driver.find_element_by_id('frameJobDescription')

If you need to get details in BOTH iframes during your script, you will need to take turns switching between frames.