How to start google cloud storage resumable upload?

Question

I'm trying to create a GCS resumable upload url on the backend, send it to the frontend to get javascript to start a big file upload directly to the bucket.

My problem, currently, is that I can't figure out how to start the upload.

All the documentation doesn't have any code, so there is nothing for me to be based on.

My code, currently, is this:

def start_resumable_upload(self):
    API_ENDPOINT = (
        'https://www.googleapis.com/upload/storage/v1beta2/'
        'b%(bucket)s/o?uploadType=resumable&name=%(object_name)s'
    )
    url_params = {
        'bucket': BUCKET,
        'object_name': self.filename
    }
    headers = {
        'X-Upload-Content-Type': self.content_type,
        'X-Upload-Content-Length': 0,
        'Content-Type': 'application/json'
    }
    r = urlfetch.fetch(
        url=API_ENDPOINT % url_params,
        method=urlfetch.POST,
        headers=headers
    )
    return r.content

start_resumable_upload is a method within the file Model I created to keep track of metadata on the database, thus, self.filename will have the filename, content_type will be the mime type and so on.

The response of that request is:

400. That’s an error.

Your client has issued a malformed or illegal request. That’s all we know.

which is somewhat unacceptable.

Any help is appreciated.

Answer 1

This sample application includes an example of doing a resumable upload in Python, using the google-api-python-client.

The critical setup is:

media = MediaFileUpload(filename, chunksize=CHUNKSIZE, resumable=True)
if not media.mimetype():
    media = MediaFileUpload(filename, DEFAULT_MIMETYPE, resumable=True)
request = service.objects().insert(bucket=bucket_name, name=object_name,
                                 media_body=media)

Answer 2

This function will generate a signed upload URL after authenticating with a file-based JSON Service Account key. This can then be returned to a client to authenticate uploads via HTTP PUT.

import json
import os
from oauth2client.service_account import ServiceAccountCredentials
import httplib2

GCP_CREDENTIALS_FILE = os.getenv('GCP_CREDENTIALS_FILE', 'client-secret.json')
GCS_UPLOAD_URL_PATTERN = 'https://www.googleapis.com/upload/storage'+ \
                         '/v1/b/{bucket}/o?uploadType=resumable'

def get_upload_url(bucket, filename, content_length, content_type='application/octet-stream',):
    credentials = ServiceAccountCredentials.from_json_keyfile_name(
        'client-secret.json',
        ('https://www.googleapis.com/auth/devstorage.read_write',),
    )
    http = httplib2.Http()
    credentials.authorize(http)
    url = GCS_UPLOAD_URL_PATTERN.format(bucket=bucket)
    body = json.dumps({
        'name': filename,
    }).encode('UTF-8')
    headers = {
        'X-Upload-Content-Type': content_type,
        'X-Upload-Content-Length': content_length,
        'Content-Type': 'application/json; charset=UTF-8',
        'Content-Length': len(body),
    }
    resp_headers, resp_body = http.request(url, method='POST', headers=headers, body=body)
    return resp_headers['location']

You could also authorize with App Engine or Compute Engine credentials. Then you would just need to change the oauth2client class you use to generate the credentials instance. If you're going to use this in production, you'll also want to add some error handling for credential problems, network issues, etc.