I have just started experimenting with PyCharm 2018.3.5, Anaconda 2018.12, but I have a lot of difficulty of setting it up with Scrapy. Therefore, I would like to ask you all for some assistance, in order to get things up and running. I am following the following tutorial in order to test whether Scrapy works with PyCharm and Anaconda. This does not seem to be the case. The setup is done on a Windows 10 machine with the latest updates.
I have also included every step of the installation process and hopefully, you would be able to point where I went wrong.
- Installing Anaconda on the default location.
- Install PyCharm on the default location.
- Install OpenSSL (for this version of PyCharm, you need to install this apparently in order for it to find packages; link: https://slproweb.com/products/Win32OpenSSL.html).
- I proceed to start PyCharm and create my first Scrapy project, using the follwing settings:
- I proceed to wait until PyCharm is done with everything and then I open Anaconda Navigator, which I will use to install scrapy and beautifulsoup4. The following image shows the result:
- In PyCharm, Scrapy is also visible:
- Next, I go into Explorer and navigate to
C:\Users\admin Man\PycharmProjects\Test(Notice that this is the same directory as in step 4). - So, first I tried using Command Prompt, but this fails:
I read that this is because the Anaconda Prompt does something with the paths. So I used the following command in Anaconda Prompt.
A small question in between. I was wondering how I can make it work in cmd.exe what I did in Anaconda prompt without activating an environment. I think answering this question would give me a better understanding in what Anaconda prompt does.
- This succeeds, now I have the following directory:
- So I just followed the tutorial online and created the following spider.
- Then I set up the following configuration
I tried to run with python console and emulate options, but that all fails. I checked whether I have the request package and I have request 2.21.0.
The error I got is the following:
C:\ProgramData\Anaconda3\envs\Test\python.exe C:/ProgramData/Anaconda3/envs/Test/Lib/site-packages/scrapy/cmdline.py crawl quotes
Traceback (most recent call last):
File "C:/ProgramData/Anaconda3/envs/Test/Lib/site-packages/scrapy/cmdline.py", line 8, in <module>
import scrapy
File "C:\ProgramData\Anaconda3\envs\Test\lib\site-packages\scrapy\__init__.py", line 27, in <module>
from . import _monkeypatches
File "C:\ProgramData\Anaconda3\envs\Test\lib\site-packages\scrapy\_monkeypatches.py", line 20, in <module>
import twisted.persisted.styles # NOQA
File "C:\ProgramData\Anaconda3\envs\Test\lib\site-packages\twisted\persisted\styles.py", line 21, in <module>
from twisted.python.compat import _PY3, _PYPY
File "C:\ProgramData\Anaconda3\envs\Test\lib\site-packages\twisted\python\__init__.py", line 11, in <module>
from .compat import unicode
File "C:\ProgramData\Anaconda3\envs\Test\lib\site-packages\twisted\python\compat.py", line 605, in <module>
from http import cookiejar as cookielib
File "C:\ProgramData\Anaconda3\envs\Test\Lib\site-packages\scrapy\http\__init__.py", line 8, in <module>
from scrapy.http.headers import Headers
File "C:\ProgramData\Anaconda3\envs\Test\lib\site-packages\scrapy\http\__init__.py", line 10, in <module>
from scrapy.http.request import Request
File "C:\ProgramData\Anaconda3\envs\Test\lib\site-packages\scrapy\http\request\__init__.py", line 8, in <module>
from w3lib.url import safe_url_string
File "C:\ProgramData\Anaconda3\envs\Test\lib\site-packages\w3lib\url.py", line 18, in <module>
from six.moves.urllib.request import pathname2url, url2pathname
File "C:\ProgramData\Anaconda3\envs\Test\lib\site-packages\six.py", line 92, in __get__
result = self._resolve()
File "C:\ProgramData\Anaconda3\envs\Test\lib\site-packages\six.py", line 160, in _resolve
module = _import_module(self.mod)
File "C:\ProgramData\Anaconda3\envs\Test\lib\site-packages\six.py", line 82, in _import_module
__import__(name)
File "C:\ProgramData\Anaconda3\envs\Test\lib\urllib\request.py", line 88, in <module>
import http.client
ModuleNotFoundError: No module named 'http.client'
Process finished with exit code 1
Does anyone how I can fix it? In addition, I also tried setting up the working directory to something else, but I don't know to where, I tried of course the project directory.
Actually, I can find the module that it seems to be missing (see below):
