Google Cloud storage on dev_appserver on Cloud Shell

Question

I am trying to use Google Cloud Storage from GAE and all works fine when deployed. I am however unable to get the storage to work on the development server that runs on Google Cloud Shell.

I am using the cloudstorage API and my understanding it that dev_appserver should access same buckets as the deployed application. For me working with local data would be fine as well.

The development server is invoked by dev_appserver.py .

I have played around with the non-documented default_gcs_bucket_name flag but no luck.

I have set he cloud shell to work with the correct project using gcloud init. I have also tried to use the new google.cloud.storage API without luck.

The same error as referenced below is triggered if I run the sample provided here: App Engine and Google Cloud Storage Sample.

Anyone?

Edit: I get the same error regardless of whether I supply a valid or an invalid bucket name.

app.yaml

runtime: python27
api_version: 1
threadsafe: true

builtins:
- remote_api: on

handlers:
- url: /.*
  script: main.app

main.py

import logging
import cloudstorage as gcs
import webapp2
from google.appengine.api import app_identity

class MainPage(webapp2.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        bucket_name = "xxxx-xxxxxx.appspot.com"     
        stats = gcs.listbucket('/'+bucket_name) 
        for stat in stats:
          self.response.write(repr(stat) + '')        

app = webapp2.WSGIApplication([
    ('/', MainPage),
], debug=True)

Log

INFO     2017-06-07 20:36:42,068 devappserver2.py:116] Skipping SDK update check.
INFO     2017-06-07 20:36:42,105 api_server.py:297] Starting API server at: http://0.0.0.0:38829
INFO     2017-06-07 20:36:42,110 dispatcher.py:209] Starting module "default" running at: http://0.0.0.0:8080
INFO     2017-06-07 20:36:42,111 admin_server.py:116] Starting admin server at: http://0.0.0.0:8000
ERROR    2017-06-07 20:36:54,836 api_server.py:374] Exception while handling service_name: "app_identity_service"
method: "GetAccessToken"
request: "\n7https://www.googleapis.com/auth/devstorage.full_control"
request_id: "QBrGUwjoJT"

Traceback (most recent call last):
  File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 349, in _handle_POST
    api_response = _execute_request(request).Encode()
  File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 225, in _execute_request
    make_request()
  File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 220, in make_request
    request_id)
  File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub.py", line 131, in MakeSyncCall
    method(request, response)
  File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/api/app_identity/app_identity_defaultcredentialsbased_stub.py", line 195, in _Dynamic_GetAccessToken
    'expires': now + token.expires_in,
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

WARNING  2017-06-07 20:36:54,838 tasklets.py:468] suspended generator _make_token_async(rest_api.py:55) raised RuntimeError(TypeError("unsupported operand type(s) for +: 'int' and 'NoneType'",))
WARNING  2017-06-07 20:36:54,839 tasklets.py:468] suspended generator get_token_async(rest_api.py:224) raised RuntimeError(TypeError("unsupported operand type(s) for +: 'int' and 'NoneType'",))
WARNING  2017-06-07 20:36:54,839 tasklets.py:468] suspended generator urlfetch_async(rest_api.py:259) raised RuntimeError(TypeError("unsupported operand type(s) for +: 'int' and 'NoneType'",))
WARNING  2017-06-07 20:36:54,839 tasklets.py:468] suspended generator run(api_utils.py:164) raised RuntimeError(TypeError("unsupported operand type(s) for +: 'int' and 'NoneType'",))
WARNING  2017-06-07 20:36:54,839 tasklets.py:468] suspended generator do_request_async(rest_api.py:198) raised RuntimeError(TypeError("unsupported operand type(s) for +: 'int' and 'NoneType'",))
WARNING  2017-06-07 20:36:54,839 tasklets.py:468] suspended generator do_request_async(storage_api.py:137) raised RuntimeError(TypeError("unsupported operand type(s) for +: 'int' and 'NoneType'",))

Check your local account using gcloud init. You may not be using the same Google account to access the Storage from prod and local. — Rad Apdal
Thanks! In cloud shell I am logged in as myslef and on GCE it is a service account. I am the owner of the project and have rights to the bucket so that should not be the problem. — Arne S
Great. I'll add an answer that includes setting permissions. — Rad Apdal
Hi Arne, Grace period ends in 3 hours, could you review my answer? Thanks — Mar Cial R

Mar Cial R Mar Cial R · Accepted Answer · 2017-06-17T08:02:42

Follow the instructions available in How the Application Default Credentials work (condition 1.)

It involves setting GOOGLE_APPLICATION_CREDENTIALS env variable to point to a valid json credentials file. For these tests, I just created a new one selecting App Engine default service account:

...and it worked beautifully.

According to How the Application Default Credentials work running gcloud auth application-default login (condition 2.) should also solve the issue but I have not been able to make it work on Cloud Shell, it fails with the same "access token related" error :-(

Google Cloud storage on dev_appserver on Cloud Shell

3 Answers