4
votes

We use the resumable upload mechanism in Google Cloud Storage to upload files from App Engine as described on: https://developers.google.com/storage/docs/json_api/v1/how-tos/upload#resumable A nice feature is that it's possible to make GCS generate upload urls from App Engine with an uploadId making it possible to use this URL directly from a client without the need to sign the URL. This works when adding the GAE service account as an GCS API project member.

This mechanism worked very well for a long time, but since today it stopped working with the following error (http 403):

"domain": "usageLimits", "reason": "accessNotConfigured", "message": "Access Not Configured. Please use Google Developers Console to activate the API for your project."

So we're a little bit lost now. Could this be related to the GCS incident (Incident Report for GCS Error Rate Spike March 4th)?

Find below the important part of the code we're using to make the first POST (which was working before):

String url = "https://www.googleapis.com/upload/storage/v1beta2/b/"+bucketName+"/o?uploadType=resumable"; connection.setRequestMethod("POST"); connection.setRequestProperty("X-Upload-Content-Type", contentType); String json = "{\"name\":\""+objectName+"\"}"; connection.setRequestProperty("Content-Length", String.valueOf(json.getBytes("UTF-8").length)); connection.setRequestProperty("Content-Type", "application/json"); List scopes = new ArrayList<>(); scopes.add("https://www.googleapis.com/auth/devstorage.full_control"); AppIdentityService.GetAccessTokenResult accessToken = AppIdentityServiceFactory.getAppIdentityService().getAccessToken(scopes); connection.setRequestProperty("Authorization", "Bearer "+accessToken.getAccessToken());

1
A few questions: does this happen 100% of the time? Do you see this on the service account side (when you call the initial POST), or do the individual clients doing PUTS see this? Also, are you particularly high volume (thousands per second)? Do other operations work?Brandon Yarbrough
Yes it happens 100% of the time and happens on the initial POST. So we're unable to get the actual upload URL needed for the PUT. We just found out the problem is actually more generic, all calls to the GCS JSON API are currently failing with this error. Things go fine if we switch to the XML GCS API, but that API doesn't offer the 'unsigned PUT'. The volume of requests isn't high (currently not much more than a couple per second).Tim Rob
I'm going to suggest that you contact [email protected] and let them know which account you're trying to use. They may be able to help you further.Brandon Yarbrough
Oh, and one other idea. Is it possible that, in the Google Cloud Console, the switch under "APIs > APIs & Auth" labeled "Google Cloud Storage JSON API" has been flipped to OFF?Brandon Yarbrough
Thanks for your help, I'll contact the gs-team. The JSON API was indeed OFF and I already tried to switch this on. That didn't help unfortunately. Until recently we only had the Google Cloud Storage switch on in the API project. But the GCS JSON API is also on now.Tim Rob

1 Answers

0
votes

To fix this we need to enable the 'JSON GCS API' on the App Engine project itself (the GCS buckets are part of a different project for which the JSON GCS API is already enabled). Unfortunately we're not able to enable the API switch, since the App Engine project is still an 'Old style' project without the APIs menu in the Google Developers Console.

To be able to enable APIs on the App Engine project we need to activate 'Cloud Integration' on the App Engine settings page. (This creates a Google Cloud project as well as a Google Cloud Storage bucket and a new style service account for this application.)

Unfortunately this is failing every time with: "An error occurred when creating the project. Please retry."

We've entered a bug for this: https://code.google.com/p/googleappengine/issues/detail?id=10906 Hopefully someone from Google is able to help us out with this..