Production App on Google App Engine suddenly lost access to Google Cloud Storage

Question

Our production app (python 2.7 standard environment) running on Google App Engine suddenly lost permissions to write (create objects) on Google Cloud Storage, without any change in the code on our side.

The code is able to create new buckets, but not new objects within them.

It seems that the default app engine service account is not granted the permission. Needless to say, the service account has the Storage Object Creator role, as well as the Editor role on the project level.

Strangely, the exact same code running on the test environment project, continues to work perfectly. We are using the api client library to obtain credentials, like so:

from oauth2client.appengine import AppAssertionCredentials
from apiclient.discovery import build as discovery_build

credentials = AppAssertionCredentials(scope=scope)
http = credentials.authorize(httplib2.Http())
service = discovery_build('storage', 'v1', http=http)

And then using the service to make the api call. All calls to create objects are suddenly failing with the message: "Anonymous caller does not have storage.objects.create access to /"

Any ideas what could suddenly have gone wrong ??

Either the OAuth Access Token has expired (which I doubt as the normal error is 403) or your code is not using the Access Token in the API call. Anonymous caller means no HTTP Authorization header. To me, this indicates a bug in your code. The Google client libraries automatically renew expired tokens. I have not used AppAssertionCredentials. Is there a reason to use this class instead of ADC (Application Default Credentials) style classes? The issue about being able to create buckets but not create objects might indicate you are using different credentials in those API calls. — John Hanley
John, Thanks for the comment, however, 1. Exactly the same code is running in the test environment without flaws (identical code, different project). 2. Bucket creation is using exactly the same credentials, and succeeds. 3. The production app was running perfectly until some hours ago, no code was changed.... — ACEGL
There is not enough information or code provided in your question for me to know what is wrong. — John Hanley
Hey, we are currently experiencing the same issue. It started on the 25.9, was working fine on the 26.9 but started to appear again on the 27.9 and it's been there since. Are you experiencing the same issues on these dates? — user3717963

ACEGL ACEGL · Accepted Answer · 2019-10-01T07:45:20

This turned up to be an issue with Google Cloud Storage (GCS). A payed support ticket was opened, after approximately 90 hours, a rollback was made by Google GCS engineers which solved the issue, however, the root cause of the issue was not found or reported. Very troubling that a production app can be affected this way for such a long time and eventually there is no explanation.

Production App on Google App Engine suddenly lost access to Google Cloud Storage

1 Answers