GCS uploadHanders extreme latency

2
votes

Is there a way to reduce the latency of GCS uploads? Can I somehow force a faster Cloud Storage for a higher rate?

If I create an uploadHandler via GAE

blobstore.create_upload_url('/upload', gs_bucket_name='bucket_name')

It creates an url like this: /_ah/upload/AMmfu6ay12yQDxuTOEjUh5...c-J09am7J4r

A small post to this url (120kb) results in this timeline: (8 Seconds latency!)

Blocking  :  2.000 ms
Sending   :  241.000 ms
Waiting   :  8.22 S
Receiving :  1.000 ms

But posting the same file to GAE itself and using the cloudstorage SDK to write the blob to GCS I get this timeline:

Blocking  :  2.000 ms
Sending   :  221.000 ms
Waiting   :  2.62 S
Receiving :  1.000 ms

Why this drastic difference? What parameters can I tweak to get the latency down? It would be great to be able to skip the extra load on my front-end GAE instances, dose anyone have experience in GCS when speed is important?

1
google-app-enginegoogle-cloud-storagegoogle-compute-engine
Your first example has blobstore sitting between App Engine and Cloud Storage. What does the latency look like with a non-gcs blobstore upload? - David
Not that bad, 2-3 sec if i remove the gcs bucket parameters - Robin Westerlundh

1 Answers

0
votes

This can be explained by thinking about the infrastructure:

  1. When you post your data to the blobstore upload URL, it is sent through the blobstore handlers, then on to GCS storage.
  2. When you post your data to blobstore with no GCS parameters, it's faster because the blob storage (from the speeds, seems to be different from GCS) is "closer" to the handler than going to GCS (make whatever inferences you will about the infrastructure based on these time differences and all known info about GCS/blobstore)
  3. When you put your data to GCS from GAE, it's much closer to where it needs to be - it's inside the data center, inside google's network. Of course you still wait for the upload to GAE.

You could look into Regional Buckets, but be aware that this is alpha and may change at any time, so probably not best to base your production on it. You can always look into support if you are experiencing an odd issue.