Accessing Cloud Storage Objects via App Engine without making bucket publicly readable

Question

I'd like to be able to access the files in a Cloud Storage Bucket from my App Engine App without making the objects or the bucket itself Publicly Readable. While I'm aware of a bunch of options out there that allow access to bucket objects with authentication (client libraries, signed urls, etc.), the complicating factor is that I'd like to be able to access the files with path that is similar to the folder structure of the bucket in question.

For example, if I make my bucket publicly readable, I can access objects with the public link: https://storage.googleapis.com/MY_BUCKET/FOLDER_IN_MY_BUCKET/FILE_IN_FOLER.txt. This url mimics the internal folder structure of the bucket. However, there doesn't appear to be a comparable url if the bucket is not publicly readable. My App Engine App service account has been added as a storage admin for the bucket I need but I'm not sure if there's a url that I can use to access the buckets objects. An object's mediaLink won't work because generation information is appended to the end, and selfLink results in a 404 error.

The need for a url like this is because the bucket contains several thousand objects. Downloading them using a client library to the the App Engine's persistent storage kind of defeats the purpose of using cloud storage in my case. Obtaining signed urls for all of them when a request is made would be time consuming and then I'd have to manage thousands of signed urls somewhere.

Is there a way to read from the cloud storage bucket with a predictable url, like the public url, while also still authenticating the request?

Answer 1

Rather than trying to vend thousands of signed URLs in the response you can create a 'redirect' endpoint in your app engine app.

e.g. user does a 'GET' against www.myapp.com/fetch/<bucket>/<object>

Your app engine code handling this endpoint authorizes the user to make sure they should have access, pulls the bucket/object out of the URL, then generates a signed URL granting access to the resource and returns a 302 redirect to that URL.

Answer 2

The URL you mention can be accessed without making the bucket or file public given that your browser is authenticated with an account having access to those resources: https://storage.cloud.google.com/MY_BUCKET/FOLDER_IN_MY_BUCK‌​ET/FILE_IN_FOLDER

Regarding the access to the file through a different application (for example App Engine), you can always use the client libraries for your preferred language. You can test how the API works in the documentation, just by defining the bucket parameter as MY_BUCKET and the object parameter as FOLDER_IN_MY_BUCKET/FILE_IN_FOLDER. You should use this same structure when applying it to the client library of your choice.