Google Cloud Datastore Python3 AttributeError on include

0
votes

I wrote a python3 scrapy spider that I have run to completion many times. Now I want to add google cloud datastore functionality to it (read/write from the datastore). I went to follow the instructions on the google docs page for this. However, when I go to include the library in my script, without even calling a single function, (just including it causes this), I get the following error:

AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'

I did a search on Stackoverflow and online, and the suggested response was to upgrade the install of the google protobuf package (as seen here:)

How to solve "AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key"?

I tried the steps outlined there to no success. I still get the same error. Keep in mind I used the python3 version of pip (pip3) to attempt my protobuf upgrade. It says it's already up to date, and says I'm using version 3.13.0. The version of python being loaded is 3.8.2. However, somebody else listed protoc --version as a command to run, but when I run that protoc is not installed and the command is not found. Keep in mind there are multiple comments regarding the correct and working version of protobuf to be 3.13.0, which pip3 already says that I have. I've attached a screenshot of the callstack in case it means something to somebody.

-

Edit: One thing I just tried is I ran the python3 interactive shell, and typed "from google.cloud import datastore", and it worked without an error. Is this possibly a clue about where the issue lies? Also when i issue the command pip3 install --upgrade google-cloud-datastore, it says all the requirements are already satisfied, including google protobuf >= 3.12.0, which is correct as its 3.13.0. The issue still occurs however.

Edit2: I also created a windows 10 vm, and the include works fine there, also with python3/anaconda. I may end up just doing my development within the vm if this is the case. It definitely seems like theres some kind of old protobuf file or dependency that is stale/wrong on my ubuntu machine.

1
pythonpython-3.xgoogle-cloud-platformincludegoogle-cloud-datastore
Seems like a caching issue or something in this order happening onto your ubuntu machine. Maybe try to delete your requirements.txt and redownload all the packages and dependencies and see if the error persists? – Stefan Neacsu
I do not see a requirements.txt file near my script. I certainly didn't create one yet. I just have a single py script file. I think what I will try next when I have the time is to set up a virtual environment for script dependencies (venv) and see if that solves it. I will also look into creating my own requirements.txt file as well, and how those two interact together. – user8694-03

1 Answers

0
votes

This problem magically resolved itself days later. As a previous commenter stated, it seemed like a caching issue, and the cache was cleaned up, and now the scrapy webcrawler works just fine.