Streaming large files (2GB+) to Google Cloud Storage bucket by way of Node/Express server

Question

I'm looking for the best way (or a way) to upload large files (1GB+) from the client side of my app to my Google Cloud Storage bucket.

The bucket is private and I'm currently trying to send the file to my node/express server and stream it to my GCS bucket. I think I'm running into issues due to file size constraints on the server though. Works fine for smaller files, but the 1GB file I'm using to test it isn't getting through.

These are the alternate methods I'm exploring:

• file splitting client side and then reassembling server-side prior to sending to GCS
• create a temporary service account through the Google IAM API (GCS write access), send this to the client to be able to upload the file, delete account after upload confirmed (not ideal since it exposes the service account)
• ???

Just looking to get pointed in the right direction at this point.

Answer 1

The most direct transfer would be from the client directly to GCS. You'd probably want to send the client a signed URL that they would use to start and then carry out the upload. You'd likely want to use a resumable upload unless you expect your customers to have fast enough Internet not to need to bother.

It would probably be a good idea to have clients upload to some staging GCS bucket and then notify your app that the upload is complete, at which point your app would then copy it to the final bucket (which is an instant operation if the objects are in the same region and storage class), although that's not necessarily required.

I would probably try to avoid streaming data through your app unless you wanted to transform it in some way or want to use multiple cloud providers transparently.

Creating one-off service accounts is probably not a great idea. I'm not sure what the limit on those is off-hand, but you may run into issues scaling that up.