Slow file uploading to Google Cloud Storage

11
votes

I'm noticing quite low upload speeds to Google Cloud Storage, almost 2.5x slower compared with uploading a file to Google Drive. Here is a screencast comparing the two for the upload of a 1GB file:

https://gyazo.com/c3488bd56b8118043b7df5aab813db01

This is just an example, but I've also tried using the gsutil command-line tool, using all the suggestions they have for uploading large files the fastest (such as using parallel_composite_upload_threshold). It is still slower than I'm accustomed to. Much slower.

Is there any way to improve this upload speed? Why is upload to Drive so much faster than doing the same to GCS?

2
google-cloud-platformgoogle-drive-apigoogle-cloud-storage
Have you checked the solution from this SO post - Slow Load Time From Google Cloud Storage Bucket? It could explain why uploading file to Google Cloud Storage is slow. – jess
@jess thanks for this. Though I'm not concerned about the download times from GCS (we never use gcs for content delivery) but the actual upload times to it. – user10332687

2 Answers

11
votes

It took me a day to upload around 30.000 images (100kb/image) using console.google.cloud.com or using browser, the same as you did. Then I tried to use gsutil to upload the files using terminal in Ubuntu. Following the instruction here : https://cloud.google.com/storage/docs/uploading-objects .

For single file :

gsutil cp [LOCAL_OBJECT_LOCATION] gs://[DESTINATION_BUCKET_NAME]/

For directory :

gsutil -m cp -R [DIR_NAME] gs://[DESTINATION_BUCKET_NAME]

Using gsutil it was incredibly faster, I uploaded 100.000 images (100~400kb/image) and it only took less than 30 minutes.

Honestly I haven't done lots of research why using gsutil is totally faster than using console. Probably because of gsutil provides the -m option which performs a parallel (multi-threaded/multi-processing) copy which can significantly increase the upload performance. https://cloud.google.com/storage/docs/composite-objects

0
votes

Well, first of all both these products serve different purposes. While Drive can be seen more of a small-scale file storage using cloud, Cloud Storage is focused in the integration with Google Cloud Platform products, data reliability, accessibility, availability in a small-to-high scale.

You need to take into account that when you are uploading a file to Cloud Storage, it is treated as a blob object, which means that it has to go through some extra steps, for example the object data needs to be encrypted when uploaded to Cloud Storage, and uploaded objects are checked for consistency.

As well, depending on the configuration of your bucket, objects uploaded might have enabled version control, and the bucket might be storing the data in various regions at the same time, which can slow the file uploads.

I believe some of this points, specially encryption, is what make file uploads slower in Cloud Storage compared to Drive.

I would recommend however, to have the bucket you are uploading to at the region closest to your area, which could make a difference.