Module import failing: How to get Google Cloud Client libraries to run on Google Cloud SDK

Question

Problem

I'm trying to develop a basic Google App Engine app. I do a lot of data science and so I use Anaconda to manage my python distributions.

Recently I've been trying to set-up the Google Cloud SDK + Google Cloud Client Libraries to develop on GAE (in the standard environment) and can't get the two to function together.

I have activated a Python 2.7 env (py27) and when I try to run a basic app I get the following error:

  File "C:\Users\dominic\Anaconda3\envs\py27\lib\site-packages\google\cloud\bigquery\__init__.py", line 31, in <module>
    from pkg_resources import get_distribution
ImportError: No module named pkg_resources

However that module exists in py27.

From the interactive console the SDK provides, I can see it's executing the correct version of python:

import sys, os
print(os.path.dirname(sys.executable))

Returns: C:\Users\dominic\Anaconda3\envs\py27

So I can't understand why it can't find that specific library. I thought it might be to do with having to install any non-third party libraries into lib (in the standard environment), as only a certain set are pre-loaded in app engine (and thus presumably the cloud SDK?), but when I try to import some of the pre-loaded libraries like flask or futures (which are also installed in this python environment, these also fail).

How do I get the Google Cloud Client libraries working in the Cloud SDK?

Other things I've tried

I've tried adding a specific version of python to my PYTHONPATH, in case it was something to do with Anaconda managing my path, however that then Python completely:

Error processing line 1 of C:\Users\dominic\Anaconda3\lib\site-packages\matplotlib-2.0.2-py3.6-nspkg.pth:

Failed to import the site module
Traceback (most recent call last):
  File "C:\Users\dominic\Anaconda3\lib\site.py", line 168, in addpackage
    exec(line)
  File "<string>", line 1, in <module>
  File "C:\Users\dominic\Anaconda3\lib\types.py", line 171, in <module>
    import functools as _functools
  File "C:\Users\dominic\Anaconda3\lib\functools.py", line 21, in <module>
    from collections import namedtuple
  File "C:\Users\dominic\Anaconda3\lib\collections\__init__.py", line 32, in <module>
    from reprlib import recursive_repr as _recursive_repr
  File "C:\Users\dominic\Anaconda3\envs\py27\Lib\site-packages\reprlib\__init__.py", line 7, in <module>
    raise ImportError('This package should not be accessible on Python 3. '
ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.

I need to keep Anaconda as my primary python installation (for job purposes), but I'd also like to be able to develop with the Cloud SDK.

Environment

• Windows 7
• Anaconda for python install.
• Python 3.6 is primary version
• I have a conda py27 environment where I'm developing.
• Google Cloud SDK installed and authenticated
• google-cloud-bigquery installed into py27

Answer 1

The google-cloud-bigquery library is not a GAE-supplied one for the standards env, so you need to vendor it into your application (GAE doesn't care if it's installed into your local python installation). Your traceback indicates it's coming from the local installation, note the py27\lib\site-packages\google\cloud\bigquery string in it.

Also to be noted here is that the standard env only supports apps designed for GAE, executed through the sandbox, not generic, standalone apps. It's unclear if the basic app you tried is of one type or the other. See import cloudstorage, ImportError: No module named google.appengine.api.

As for flask or other GAE-provided libraries you need to request them in your app.yaml file (personally I'd use an explicit version in there instead of latest, I think I saw some issues reported possibly related to latest). To be noted here that some of these libraries/versions aren't installed by default in the cloud SDK, the additional app-engine-python-extras component may need to be installed, see PyCharm - Can't create App Engine Application using DJango.

As for your last attempt:

• just in case it's not clear, note that the standard env only supports python 2.7, python 3 won't work. The This package should not be accessible on Python 3 string could indicate the attempt may have been done using python 3. You'll need to explicitly use python 2.7 executables since your primary version is python 3.
• matplotlib is one of the few GAE-provided libs which needs extra care for the local development using the SDK, see Using built-in bundled libraries with the local development server