Invalid credentials - 401 when trying to access Google Datastore from a GCE instance

Question

I'm trying the adams.py example found on this URL https://cloud.google.com/datastore/docs/getstarted/start_python/

As it's explained in this URL:

If you are running this code on Google Compute Engine, you do not have to create new service account credentials because the default service account for your project is used when you create an instance with the datastore and userinfo-email scopes.

While creating my instance, I checked the option:

Allow API access to all Google Cloud services in the same project.

When I run

python adams.py my-instance

I get:

    ERROR:root:Error while doing datastore operation
    ERROR:root:RPCError: beginTransaction Invalid Credentials
    ERROR:root:HTTPError: 401 Unauthorized

In fact I get the same message even if I use a wrong name for the instance.

My problem seems similar to this thread from one year ago: Connect google datastore from existing google compute engine in python

In my own code, it works fine accessing BigQuery and then when it comes to Datastore it raises the same error, but at least is more specific addressing the line, which is the last one: resp = datastore.commit(req). Here is the section of the code use Datastore.

req = datastore.CommitRequest()
req.mode = datastore.CommitRequest.NON_TRANSACTIONAL
dsDashboard = req.mutation.insert_auto_id.add()

path_element = dsDashboard.key.path_element.add()
path_element.kind = 'Dashboard'
fmt = '%Y-%m-%d'
dashboardDate = dsDashboard.property.add()
dashboardDate.name = 'Date'
dashboardDate.value.string_value = dtDay.strftime(fmt)

dashboardValue = dsDashboard.property.add()
dashboardValue.name = 'Value'
dashboardValue.value.indexed = False
dashboardValue.value.string_value = encode(jDashboard)

resp = datastore.commit(req)

The error for my code:

    Traceback (most recent call last):
      File "code.py", line 262, in <module>
        main()
      File "code.py", line 256, in main
        resp = datastore.commit(req)
      File "/usr/local/lib/python2.7/dist-packages/googledatastore/__init__.py", line 90, in commit
        return get_default_connection().commit(request)
      File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 135, in commit
        datastore_v1_pb2.CommitResponse)
      File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 195, in _call_method
        raise RPCError(method, response, content)
    googledatastore.connection.RPCError: commit RPC client failure with HTTP(401) Unauthorized: Invalid Credentials

In short, it seems something simple to do. Anyone here had the same problem and has found a workaround?

Hi. Thank you! Before your answer I had to use a workaround. I was curious about your solution but I didn't have time to test. Thanks anyway! — marioishikawa

Mar Cial R Mar Cial R · Accepted Answer · 2015-12-13T11:19:12

The issue here is:

While creating my instance, I checked the option:

Allow API access to all Google Cloud services in the same project.

One would assume that this would make your instance get all scopes but It seems it is not working so. I've tried it and if you inspect a newly created instance with:

gcloud compute instances describe my-instance

you'll see that it only gets one scope

scopes:
  - https://www.googleapis.com/auth/cloud-platform

The solution here is to either

manually enable "User info" and "Cloud Datastore" ("Management, disk, networking, access & security options" > "Access & security") when creating your instance via Developers console
make sure you use correct scopes flag (--scopes datastore,userinfo-email) when using gcloud command as explained in Getting started with Google Cloud Datastore and Python/Protobuf article

Invalid credentials - 401 when trying to access Google Datastore from a GCE instance

1 Answers