Direct user upload to GCS with given name and parameters

Question

I want to enable clients of my Python GAE app to upload files directly to GCS using a specific object name and GCS parameters I set up front (such as cache control and ACL), and perform some actions after the upload has finished.

I currently use the blobstore.create_upload_url for this. However, since I cannot give any parameters or object name up front, I need to copy the entire file to a new location on GCS from within the 'uploaded' callback handler. This is time and (compute) resource consuming, and sounds like it could be avoided.

Is there any way to optimize this process?

Can I create an upload URL that will immediately upload an object with the right name and right ACL + Cache control parameters? (I briefly looked at signed URLs, but those don't seem to support a callback handler like the blobstore API does)
If not, is there a way to not have to copy the file byte by byte to the new location and delete it when i'm done? (being able to move a file and change its attributes would be nice)

Update: I just noticed the GCS python client library has copy2 now. I could have sworn this wasn't there before, but it sounds like it answers the second question.

Yes you can (without copy) using a post object form: cloud.google.com/storage/docs/reference-methods#postobject — voscausa
@voscausa This does not seem to do anything similar to create_upload_url? I want to give a (signed) url to the client such that he can upload somewhere, without needing any credentials for GCS or whatever (so handling authorization server-side using my own logic in the request to get an URL) — Remko
You create a signed form. See the last line in the form: <input type="hidden" name="signature" value".....">. — voscausa

voscausa voscausa · Accepted Answer · 2015-07-17T05:00:14

Here is my Python, webapp2, jinja code for a direct upload (post) to GCS using a signed form with a policy document:

def gcs_upload(acl='bucket-owner-read'):
    """ jinja2 upload form context """

    default_bucket = app_identity.get_default_gcs_bucket_name()
    google_access_id = app_identity.get_service_account_name()
    succes_redirect = webapp2.uri_for('_gcs_upload_ok', _full=True)
    expiration_dt = datetime.now() + timedelta(seconds=300)

    policy_string = """
    {"expiration": "%s",
              "conditions": [
                  ["starts-with", "$key", ""],
                  {"acl": "%s"},
                  {"success_action_redirect": "%s"},
                  {"success_action_status": "201"},
              ]}""" % (expiration_dt.replace(microsecond=0).isoformat() + 'Z', acl, succes_redirect)

    policy = base64.b64encode(policy_string)
    _, signature_bytes = app_identity.sign_blob(policy)
    signature = base64.b64encode(signature_bytes)

    return dict(form_bucket=default_bucket, form_access_id=google_access_id, form_policy=policy, form_signature=signature,
            form_succes_redirect=succes_redirect, form_acl=acl)


class GcsUpload(BaseHandler):

    def get(self):

        self.render_template('gcs_upload.html', **gcs_upload())

And the form fields:

<form method="post" action="https://storage.googleapis.com/{{ form_bucket }}" enctype="multipart/form-data">
    <input type="hidden" id="key" name="key" value="images/test.png">
    <input type="hidden" name="GoogleAccessId" value="{{ form_access_id }}">
    <input type="hidden" name="acl" value="{{ form_acl }}">
    <input type="hidden" name="success_action_redirect" value="{{ form_succes_redirect }}">
    <input type="hidden" name="success_action_status" value="201">
    <input type="hidden" name="policy" value="{{ form_policy }}">
    <input type="hidden" name="signature" value="{{ form_signature }}">
    <input name="file" type="file">
    <input type="submit" value="Upload">
</form>

Docs GCS upload using a form here

Direct user upload to GCS with given name and parameters

1 Answers