Upload large files to Google Cloud Storage using Google App Engine

Question

I would like to upload files up to 1GB to Google Cloud Storage. I'm using Google App Engine Flexible. From what I understand, GAE has a 32MB limit on file uploads, which means I have to either upload directly to GCS or break the file into chunks.

This answer from several years ago suggests using the Blobstore API, however there doesn't seem to be an option for Node.js and the documentation also recommends using GCS instead of Blobstore to store files.

After doing some searching, it seems like using signed urls to upload directly to GCS may be the best option, but I'm having trouble finding any example code on how to do this. Is this the best way and are there any examples of how to do this using App Engine with Node.js?

Answer 1

Your best bet would be to use the Cloud Storage client library for Node.js to create a resumable upload.

Here's the official code example on how to create the session URI:

const {Storage} = require('@google-cloud/storage');
const storage = new Storage();
const myBucket = storage.bucket('my-bucket');

const file = myBucket.file('my-file');
file.createResumableUpload(function(err, uri) {
  if (!err) {
    // `uri` can be used to PUT data to.
  }
});

//-
// If the callback is omitted, we'll return a Promise.
//-
file.createResumableUpload().then(function(data) {
  const uri = data[0];
});

Edit: It seems you can nowadays use the createWriteStream method to perform an uplaod without having to worry about creation of a URL.