Google Cloud App Engine - gsutil access issue

Question

I am looking for help to troubleshoot my application written in Python3/Flask to be hosted on Google App Engine. The deployment succeeds, but there are issues with App Engine not able to access GCS.

The code is as below:

gs_util_command = "gsutil "
gs_util_command =  gs_util_command + " cp gs://"+ re.match(r"https.//storage.googleapis.com/(.*)",url1).group(1)
gs_util_command = gs_util_command + " "+str(destination_file_path)

I keep receiving the error message as below

Error messages: ServiceException: 401 Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.

The IAM confirms that the service has owner access to the GCS storage bucket, but it still does not work.

I tried running the below statement inside the Python3 code and deploy this to GCP:

  os.system("gsutil version -l")

I get the below message in the shell log after deployment and execution on cloud:

2021-04-12 01:36:18 default[20210412t013030] gsutil version: 4.61 2021-04-12 01:36:18 default[20210412t013030] checksum: PACKAGED_GSUTIL_INSTALLS_DO_NOT_HAVE_CHECKSUMS (!= 61834ed701604b43c83de6cfae9b36a4) 2021-04-12 01:36:18 default[20210412t013030] boto version: 2.49.0 2021-04-12 01:36:18 default[20210412t013030] python version: 3.7.10 (default, Mar 23 2021, 05:09:08) [GCC 7.5.0] 2021-04-12 01:36:18 default[20210412t013030] OS: Linux 4.4.0 2021-04-12 01:36:18 default[20210412t013030] multiprocessing available: True 2021-04-12 01:36:18 default[20210412t013030] using cloud sdk: False 2021-04-12 01:36:18 default[20210412t013030] pass cloud sdk credentials to gsutil: False 2021-04-12 01:36:18 default[20210412t013030] config path(s): No config found

When i am trying to run the same command locally through Google Cloud Shell, it works fine.

gsutil version -l

The output is:

gsutil version: 4.60 checksum: 4e34e1f62b6d54d4826625581f787044 (OK) boto version: 2.49.0 python version: 3.7.3 (default, Jan 22 2021,
20:04:44) [GCC 8.3.0] OS: Linux 5.4.89+ multiprocessing available: True using cloud sdk: True pass cloud sdk credentials to gsutil: True config path(s): /home/jobsage_sai/.boto gsutil path: /usr/lib/google-cloud-sdk/bin/gsutil compiled crcmod: False installed via package manager: False editable install: False

All of this is within the virtual environment.

Please suggest.

Arvind R.

guillaume blaquiere guillaume blaquiere · Accepted Answer · 2021-04-12T11:18:51

When you deploy your code on App Engine standard, you deploy only your code. It is taken by Google Cloud, a container is created (with Buildpacks) and then it is deployed.

When you create an app with App Engine it already creates a default bucket and connections to the GCS. You don't need to set up anything else in your project to be able to use

Note that : The "container is built by Google Cloud". That means, you don't control the execution environment, what is present on not on the base image, bellow you own code. Maybe today gsutil is installed, but nothing guaranteed in the future!

So, as Dishant said, use the client libraries, or directly the APIs to achieve what you want.

If you use App Engine Flex, or Cloud Run, it's different. In both cases, you can deploy your own container (with a custom runtime on app Engine flex). And thus you can rely on the underlying runtime because you choose and manage it yourselves.

Google Cloud App Engine - gsutil access issue

2 Answers