I am new to Scrapy and only have a basic understanding of python.
Running into an error when running the Scrapy tutorial.
I have a Windows 7 OS, 64 bit and a 32 bit Python 2.7.12 installed
Steps I've taken so far
- Followed steps as per the installation page and set up a virtual environment 'scrapy'
- activated the virtual environment
- Installed scrapy (all the dependencies too seem to have installed fine, including pywin, the 32 bit version)
- Ran steps as per the tutorial page
scrapy startproject tutorial- created file
quotes_spider.py, with the code given on the tutorial page From scrapy\Scripts\tutorial I ran
scrapy crawl quotesand got the following error(scrapy) F:\My Projects\scrapy\Scripts\tutorial>scrapy crawl quotes 2016-11-05 14:21:59 [scrapy] INFO: Scrapy 1.2.1 started (bot: tutorial) 2016-11-05 14:21:59 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tutorial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'ROBOTSTXT_OBEY': True, 'BOT_NAME': 'tutorial'} 2016-11-05 14:21:59 [scrapy] INFO: Enabled extensions:'scrapy.extensions.logstats.LogStats','scrapy.extensions.telnet.Telne tConsole', 'scrapy.extensions.corestats.CoreStats'] Unhandled error in Deferred: 2016-11-05 14:21:59 [twisted] CRITICAL: Unhandled error in Deferred: 2016-11-05 14:21:59 [twisted] CRITICAL: Traceback (most recent call last): File "f:\myproj~1\scrapy\lib\site-packages\twisted\internet\defer.py", line 12 60, in _inlineCallbacks result = g.send(result) File "f:\myproj~1\scrapy\lib\site-packages\scrapy\crawler.py", line 90, in crawl six.reraise(*exc_info) File "f:\myproj~1\scrapy\lib\site-packages\scrapy\crawler.py", line 72, in crawl self.engine = self._create_engine() File "f:\myproj~1\scrapy\lib\site-packages\scrapy\crawler.py", line 97, in _create_engine return ExecutionEngine(self, lambda _: self.stop()) File "f:\myproj~1\scrapy\lib\site-packages\scrapy\core\engine.py", line 68, in __init__ self.downloader = downloader_cls(crawler) File "f:\myproj~1\scrapy\lib\site- packages\scrapy\core\downloader\__init__.py" , line 88, in __init__ self.middleware = DownloaderMiddlewareManager.from_crawler(crawler) File "f:\myproj~1\scrapy\lib\site-packages\scrapy\middleware.py", line 58, in from_crawler return cls.from_settings(crawler.settings, crawler) File "f:\myproj~1\scrapy\lib\site-packages\scrapy\middleware.py", line 34, in from_settings mwcls = load_object(clspath) File "f:\myproj~1\scrapy\lib\site-packages\scrapy\utils\misc.py", line 44, in load_object mod = import_module(module) File "c:\python27\Lib\importlib\__init__.py", line 37, in import_module__import__(name) File "f:\myproj~1\scrapy\lib\site- packages\scrapy\downloadermiddlewares\retry.py", line 23, in <module> from scrapy.xlib.tx import ResponseFailed File "f:\myproj~1\scrapy\lib\site-packages\scrapy\xlib\tx\__init__.py", line 3, in <module> from twisted.web import client File "f:\myproj~1\scrapy\lib\site-packages\twisted\web\client.py", line 42, in <module> from twisted.internet.endpoints import TCP4ClientEndpoint, SSL4ClientEndpoint File "f:\myproj~1\scrapy\lib\site-packages\twisted\internet\endpoints.py", line 36, in <module> from twisted.internet.stdio import StandardIO, PipeAddress File "f:\myproj~1\scrapy\lib\site-packages\twisted\internet\stdio.py", line 30, in <module> from twisted.internet import _win32stdio File "f:\myproj~1\scrapy\lib\site- packages\twisted\internet\_win32stdio.py", line 9, in <module> import win32api ImportError: No module named win32api
This is quite similar to the error discussed here but the solutions suggested don't seem to work out. pywin is installed in my global environment C:Python27\Lib\site-packages but it doesn't show up in my virtual environment. I'm not sure how to install it in my virtual env since the pywin executable file only seems to allow installation under C. Thanks in advance for your help!
My pip list in the virtual env is
attrs==16.2.0
cffi==1.8.3
constantly==15.1.0
cryptography==1.5.2
cssselect==1.0.0
enum34==1.1.6
idna==2.1
incremental==16.10.1
ipaddress==1.0.17
lxml==3.6.4
parsel==1.0.3
pyasn1==0.1.9
pyasn1-modules==0.0.8
pycparser==2.17
PyDispatcher==2.0.5
pyOpenSSL==16.2.0
queuelib==1.4.2
Scrapy==1.2.1
service-identity==16.0.0
six==1.10.0
Twisted==16.5.0
w3lib==1.15.0
zope.interface==4.3.2
