How do I create a Google Cloud Storage resumable upload URL with Google Client Library for Java on App Engine?

Question

I found the follow note, which describes exactly what I'd like to do:

Note: If your users are only uploading resources (writing) to an access-controlled bucket, you can use the resumable uploads functionality of Google Cloud Storage, and avoid signing URLs or requiring a Google account. In a resumable upload scenario, your (server-side) code authenticates and initiates an upload to Google Cloud Storage without actually uploading any data. The initiation request returns an upload ID, which can then be used in a client request to upload the data. The client request does not need to be signed because the upload ID, in effect, acts as an authentication token. If you choose this path, be sure to transmit the upload ID over HTTPS.

https://cloud.google.com/storage/docs/access-control#Signed-URLs

However, I cannot figure out how to do this with the Google Cloud Storage Library for Java.

https://developers.google.com/resources/api-libraries/documentation/storage/v1/java/latest/

I can't find any reference to resumable files, or getting the URL for a file anywhere in this API. How can I do this?

Answer 1

That library does not expose the URLs that it creates to its caller, which means you can't use it to accomplish this. If you want to use either signed URLs or the trick you mention above, you'll need to implement it manually.

I would advise going with the signed URL solution over the solution where the server initializes the resumable upload, if possible. It's more flexible and easier to get right, and there are some odd edge cases with the latter method that you could run into.

Someone wrote a up a quick example of signing a URL from App Engine a while back in another question: Cloud storage and secure download strategy on app engine. GCS acl or blobstore

Answer 2

You can build the url yourself. Here is an example :

OkHttpClient client = new OkHttpClient();
AppIdentityService appIdentityService = credential.getAppIdentityService();
Collection<String> scopes = credential.getScopes();
String accessToken = appIdentityService.getAccessToken(scopes).getAccessToken();
Request request = new Request.Builder()
        .url("https://www.googleapis.com/upload/storage/v1/b/" + bucket + "/o?name=" + fileName + "&uploadType=resumable")
        .post(RequestBody.create(MediaType.parse(mimeType), new byte[0]))
        .addHeader("X-Upload-Content-Type", mimeType)
        .addHeader("X-Upload-Content-Length", "" + length)
        .addHeader("Origin", "http://localhost:8080")
        .addHeader("Origin", "*")
        .addHeader("authorization", "Bearer "+accessToken)
        .build();
Response response = client.newCall(request).execute();
return response.header("location");

Answer 3

It took some digging, but I came up with the following which does the right thing. Some official documentation on how to do this would have been nice, especially because the endpoint for actually triggering the resumable upload is different from what the docs call out. What is here came from using the gsutil tool to sign requests and then working out what was being done. The under-documented additional thing is that the code which POSTs to this URL to get a resumable session URL must include the "x-goog-resumable: start" header to trigger the upload. From there, everything is the same as the docs for performing a resumable upload to GCS.

import base64
import datetime
import time
import urllib

from google.appengine.api import app_identity

SIGNED_URL_EXPIRATION = datetime.timedelta(days=7)

def SignResumableUploadUrl(gcs_resource_path):
  """Generates a signed resumable upload URL.

  Note that documentation on this ability is sketchy. The canonical source
  is derived from running the gsutil program to generate a RESUMABLE URL
  with the "-m RESUMABLE" argument. Run "gsutil help signurl" for info and
  the following for an example:
    gsutil -m RESUMABLE -d 10m keyfile gs://bucket/file/name

  Note that this generates a URL different from the standard mechanism for
  deriving a resumable start URL and the initiator needs to add the header:
    x-goog-resumable:start

  Args:
    gcs_resource_path: The path of the GCS resource, including bucket name.

  Returns:
    A full signed URL.
  """
  method = "POST"
  expiration = datetime.datetime.utcnow() + SIGNED_URL_EXPIRATION
  expiration = int(time.mktime(expiration.timetuple()))
  signature_string = "\n".join([
      method,
      "",  # content md5
      "",  # content type
      str(expiration),
      "x-goog-resumable:start",
      gcs_resource_path
  ])
  _, signature_bytes = app_identity.sign_blob(signature_string)
  signature = base64.b64encode(signature_bytes)

  query_params = {
      "GoogleAccessId": app_identity.get_service_account_name(),
      "Expires": str(expiration),
      "Signature": signature,
  }

  return "{endpoint}{resource}?{querystring}".format(
      endpoint="https://storage.googleapis.com",
      resource=gcs_resource_path,
      querystring=urllib.urlencode(query_params))