Serving large file from Google Cloud Storage in Google App Engine

Question

Running a dart server in App Engine Flexible Environment, there seems to be a limitation of serving files larger than 32MB.

There are a few requirements to files I want to serve:

file size can be larger than 32MB
can not be publicly accessible (authorization is done on the server)

At the moment I try to read the file from the bucket using the gcloud library and then pipe into the request.response. This fails because of the limitation e.g.: HTTP response was too large: 33554744. The limit is: 33554432.

Is there a way to serve larger files from storage? The documentation on this topic is quite confusing (I don't think there is dart specific documentation at all). I keep reading something about the Blobstore but I am not sure if this solution is applicable for dart.

FWIW, I don't think this is Dart specific at all. The 32MB size limitation applies to all of App Engine FE. BlobStore is one option, but it looks like Cloud Storage succeeds it (cloud.google.com/appengine/docs/standard/python/blobstore, first note). Are you looking for something like github.com/GoogleCloudPlatform/google-cloud-python/blob/master/… but in Dart? — filiph
Is cloud.google.com/storage/docs/access-control/signed-urls relevant here? If so, would having that in the gcloud lib API help you? — filiph
@filiph It would be really convenient if I could do that directly from the Dart gcloud API. At the moment I use my own implementation using googleapis_auth/src/crypto/pem.dart and googleapis_auth/src/crypto/rsa_sign.dart. — Martin Flucka
@filiph we created an issue for this in the gcloud repository — Martin Flucka

Torstein I. Bø Torstein I. Bø · Accepted Answer · 2018-07-05T16:14:25

As @filiph suggest you can use signed urls from Google Cloud Storage.

Server side I have this code in python:

import time
import base64

from oauth2client.service_account import ServiceAccountCredentials


def create_signed_url(file_location):
    # Get credentials
    creds = ServiceAccountCredentials.from_json_keyfile_name('filename_of_private_key.json')
    client_id = creds.service_account_email

    # Set time limit to two hours from now
    timestamp = time.time() + 2 * 3600

    # Generate signature string
    signature_string = "GET\n\n\n%d\n%s" % (timestamp, file_location)
    signature = creds.sign_blob(signature_string)[1]
    encoded_signature = base64.b64encode(signature)
    encoded_signature = encoded_signature.replace("+", "%2B").replace("/", "%2F")

    # Generate url
    return "https://storage.googleapis.com%s?GoogleAccessId=%s&Expires=%d&&Signature=%s" % (
    file_location, client_id, timestamp, encoded_signature)

Serving large file from Google Cloud Storage in Google App Engine

1 Answers