Using Scrapy and send_keys on the following input field

Question

I am trying to focus on the email input on twitter.com/login and send_keys('foo'), however I am having no luck. Can someone tell me why my implementation is wrong:

import time

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector

from selenium import webdriver 

class MySpider(BaseSpider):
    name = "new"
    allowed_domains = ["twitter.com"]
    start_urls = ["https://www.twitter.com/login"]

    def __init__(self):
    self.driver = webdriver.Firefox()

    def parse(self, response):
    self.driver.get(response.url)
    time.sleep(1)

    #print response.body
    username = self.driver.find_element_by_class_name('email-input')
    username.click()
    username.clear()
    username.send_keys('foo')


    #self.driver.close()

Any help would be appreciated. Also, I get the following error when I attempt this:

2015-08-22 15:44:09-0500 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 2015-08-22 15:44:10-0500 [new] DEBUG: Redirecting (301) to https://twitter.com/login> from https://www.twitter.com/login> 2015-08-22 15:44:10-0500 [new] DEBUG: Crawled (200) https://twitter.com/login> (referer: None) 2015-08-22 15:44:22-0500 [new] ERROR: Spider error processing https://twitter.com/login> (referer: None) Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop self.runUntilCurrent() File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent call.func(*call.args, **call.kw) File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback self._startRunCallbacks(result) File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks self._runCallbacks() --- --- File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/home/tyrick/Documents/twitter_followers/twitter_followers/spiders/newUsers.py", line 22, in parse username.click() File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 65, in click self._execute(Command.CLICK_ELEMENT) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 385, in _execute return self._parent.execute(command, params) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 173, in execute self.error_handler.check_response(response) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 166, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.ElementNotVisibleException: Message: Element is not currently visible and so may not be interacted with Stacktrace: at fxdriver.preconditions.visible (file:///tmp/tmpHPD9xT/extensions/[email protected]/components/command-processor.js:8959:12) at DelayedCommand.prototype.checkPreconditions_ (file:///tmp/tmpHPD9xT/extensions/[email protected]/components/command-processor.js:11618:15) at DelayedCommand.prototype.executeInternal_/h (file:///tmp/tmpHPD9xT/extensions/[email protected]/components/command-processor.js:11635:11) at DelayedCommand.prototype.executeInternal_ (file:///tmp/tmpHPD9xT/extensions/[email protected]/components/command-processor.js:11640:7) at DelayedCommand.prototype.execute/< (file:///tmp/tmpHPD9xT/extensions/[email protected]/components/command-processor.js:11582:5)

alecxe alecxe · Accepted Answer · 2015-08-22T20:54:38

The problem is, there are multiple elements with email-input class, and the first one, that is actually located, is invisible. You need the one in the page container:

self.driver.find_element_by_css_selector("#page-container .email-input")

Using Scrapy and send_keys on the following input field

1 Answers