Comparison of uploading files to GCS using Google Drive vs gsutil

Question

I have been comparing how to upload files to a cloud storage, one is in-browser (or emulating a browser) and the other is command-line via gsutil to a Google Cloud Storage bucket.

Does Google Drive use gsutil in the backend, or or the uploader a totally customized and proprietary piece of software? Is there a way to achieve upload speeds to a Google Cloud Storage bucket similar to the upload speeds I'm able to achieve via Drive? If not, what would you suggest for how to get upload speeds equivalent to that in Google Drive, to upload files to a GCS bucket?

Answer 1

I'm not sure about GDrive using gsutil on the background. There are several optimizations that you can use to improve gsutil speeds.

First of all you might use perfdiag to launch a small diagnostics tests that will give you and overview and possible speeds achievable.

gsutil perfdiag -o test.json gs://<your bucket name>

Secondly you will need to understand your workload(small/big files) and identifying the need for a regional or multi regional bucket(yes there is a perf difference)tl;dr:

"Regional buckets are great for data processing since their physical distance is fairly tight, and the overhead of write consistency is low."

"Multiregional Storage, on the other hand, guarantees 2 replicates which are geo diverse (100 miles apart) which can get better remote latency and availability. "

There is some information on cloud Atlas specifically on this topic, you can check out in here:

https://medium.com/google-cloud/google-cloud-storage-what-bucket-class-for-the-best-performance-5c847ac8f9f2

https://medium.com/google-cloud/google-cloud-storage-large-object-upload-speeds-7339751eaa24?source=user_profile---------12----------------

https://medium.com/@duhroach/optimizing-google-cloud-storage-small-file-upload-performance-ad26530201dc

https://medium.com/@duhroach/google-cloud-storage-performance-4cfcec8bad72

https://cloud.google.com/storage/docs/best-practices