I tried to use scrapy_user_agents with scrapy-proxy-pool.
I added these lines in my settings.py:
DOWNLOADER_MIDDLEWARES = {
'scrapy_proxy_pool.middlewares.ProxyPoolMiddleware': 610,
'scrapy_proxy_pool.middlewares.BanDetectionMiddleware': 620,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 700,
}
when I run my spider, I get this error message:
ModuleNotFoundError: No module named 'scrapy_user_agents'
I removed the lines of proxy in the middleware, but I get same issue, same error message.
You will find below the complete log errors:
2019-08-13 16:05:28 [scrapy.utils.log] INFO: Scrapy 1.7.3 started (bot: scraping_entreprises) 2019-08-13 16:05:28 [scrapy.utils.log] INFO: Versions: lxml 4.4.1.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.7.0, Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 19.0.0 (OpenSSL 1.1.1c 28 May 2019), cryptography 2.7, Platform Windows-10-10.0.17134-SP0 2019-08-13 16:05:28 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'scraping_entreprises', 'NEWSPIDER_MODULE': 'scraping_entreprises.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MOD ULES': ['scraping_entreprises.spiders']} 2019-08-13 16:05:28 [scrapy.extensions.telnet] INFO: Telnet Password: 0a7932c1a3ce188f 2019-08-13 16:05:28 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.logstats.LogStats'] Unhandled error in Deferred: 2019-08-13 16:05:29 [twisted] CRITICAL: Unhandled error in Deferred: Traceback (most recent call last): File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\crawler.py", line 184, in crawl return self._crawl(crawler, *args, **kwargs) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\crawler.py", line 188, in _crawl d = crawler.crawl(*args, **kwargs) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\twisted\internet\defer.py", line 1613, in unwindGenerator return _cancellableInlineCallbacks(gen) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\twisted\internet\defer.py", line 1529, in _cancellableInlineCallbacks _inlineCallbacks(None, g, status)--- --- File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks result = g.send(result) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\crawler.py", line 86, in crawl self.engine = self._create_engine() File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\crawler.py", line 111, in create_engine return ExecutionEngine(self, lambda : self.stop()) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\core\engine.py", line 69, in init self.downloader = downloader_cls(crawler) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\core\downloader_init.py", line 86, in init self.middleware = DownloaderMiddlewareManager.from_crawler(crawler) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\middleware.py", line 53, in from_crawler return cls.from_settings(crawler.settings, crawler) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\middleware.py", line 34, in from_settings mwcls = load_object(clspath) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\utils\misc.py", line 46, in load_object mod = import_module(module) File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\importlib_init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 953, in _find_and_load_unlocked File "", line 219, in _call_with_frames_removed File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 965, in _find_and_load_unlocked builtins.ModuleNotFoundError: No module named 'scrapy_user_agents' 2019-08-13 16:05:29 [twisted] CRITICAL: Traceback (most recent call last): File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks result = g.send(result) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\crawler.py", line 86, in crawl self.engine = self._create_engine() File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\crawler.py", line 111, in create_engine return ExecutionEngine(self, lambda : self.stop()) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\core\engine.py", line 69, in init self.downloader = downloader_cls(crawler) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\core\downloader_init.py", line 86, in init self.middleware = DownloaderMiddlewareManager.from_crawler(crawler) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\middleware.py", line 53, in from_crawler return cls.from_settings(crawler.settings, crawler) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\middleware.py", line 34, in from_settings mwcls = load_object(clspath) File "c:\users\nino\pycharmprojects\scraping\venv\lib\site-packages\scrapy\utils\misc.py", line 46, in load_object mod = import_module(module) File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\importlib_init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 953, in _find_and_load_unlocked File "", line 219, in _call_with_frames_removed File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 965, in _find_and_load_unlocked ModuleNotFoundError: No module named 'scrapy_user_agents'
scrapy-user-agentsinstalled? – tgikal