Selenium - identifying web element

2
votes

I am using Python to scrape data from a website. While I have been able to use Selenium to log in, I cannot identify the search field once logged in. It appears the web page loads with frames (not iframes), but I cannot access the frame with the search field.

I have tried changing the frame to the relevant frame (which seems to work - no error is thrown up) but then if I try searching for the search element by CSS / Xpath / Name / id I get a NoSuchElementException. I am using the Chrome webdriver.

Any suggestions? The page source is as follows:

  <html>
  <head>
    <title> XYZ </title>
    <meta http-equiv="content-type" content="text/html; charset=utf-8" />
    <meta http-equiv="Content-Script-Type" content="text/javascript" />
    <meta http-equiv="Content-Style-Type" content="text/css" />
    <meta http-equiv="content-language" content="en" />
    <script type="text/javascript">
    if (navigator && navigator.appVersion && navigator.appVersion.match("Safari") && !navigator.appVersion.match("Chrome")) {
        // hack to force a window redraw
        window.onload = function() {
        document.getElementsByTagName('html')[0].style.backgroundColor = '#000000';
        }
    }
    </script>
  </head>

  <frameset id="wc-frameset" rows="82,*" frameborder="no" border="0" framespacing="0">
    <frame frameborder="0" src="/frontend/header/" name="top" marginwidth="0" marginheight="0" scrolling="no" noresize="noresize" />
      <frameset cols="*,156,850,*" frameborder="NO" border="0" framespacing="0">
        <frame frameborder="0" src="/frontend/fillbar/" name="fillbar" marginwidth="0" marginheight="0" scrolling="no"  noresize="noresize" />
        <frame frameborder="0" src="/frontend/navigation/" name="navigation" marginwidth="0" marginheight="0" scrolling="no"  noresize="noresize" />
        <frame frameborder="0" src="/frontend/frames/" name="content_area" marginwidth="0" marginheight="0" scrolling="no" noresize>
        <frame frameborder="0" src="/frontend/fillbar/" name="fillbar" marginwidth="0" marginheight="0" scrolling="no"  noresize="noresize" />
      </frameset>
  </frameset>
</html>

The code that I have so far is:

username = driver.find_element_by_id("username")
password = driver.find_element_by_id("password")
username.send_keys("****")
password.send_keys("****")   
driver.find_element_by_class_name("bg-left").click()
#this bit works

driver.switch_to_frame("content_area")
#this seems to work too, got the frame name from the page source

search = driver.find_element_by_id("field-name")
search.send_keys("TEST")
#this fails, no element found

The target frame source code is:

      <div id="field-name" class="field field-StringField">
  <label for="name">Name</label>            <div class="input-con"><input id="name" name="name" type="text" value=""></div>
      </div>
2
pythoncssseleniumxpathweb-scraping
Am I wrong? When you search for the id 'field-name' you get a div element but you want to send keystrokes to an input element. Wouldn't that be the element with id 'name'? - Bill Bell
I've tried it that way as well, but still got an exception:selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"name"} - Owais Arshad
You mean that name = driver.find_element_by_id("name") raised an exception? - Bill Bell
Yes using 'name' rather than 'field-name' still causes an exception - Owais Arshad

2 Answers

1
votes

It is possible that there are duplicate elements in the page. Try the following in chrome:

  1. Open url in chrome
  2. Open developer tools F12
  3. Press ESC to open the chrome console
  4. Select your frame enter image description here
  5. Search for similar elements using xpath in console $x("//input[@id='name']")

This should list the number of elements.

1
votes

Maybe you need to wait for the page to load up completely before continuing searching the element. You can try something like:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

driver.switch_to_frame("content_area")

try:
    # this line adds wait for the element to be visible 
    WebDriverWait(driver, 10).until(EC.visibility_of_element_located(By.ID, 'name'))
except TimeoutException:
    # display page timed out error

search = driver.find_element_by_id("name")
search.send_keys("TEST")