How to upload milions object to google cloud storage

1
votes

Google cloud storage API (XML or Json ) only permit 1 upload per second .

https://cloud.google.com/storage/docs/concepts-techniques

You can update a particular object up to once per second. Updating a particular object faster than once per second may result in 503 Service Unavailable errors. Concurrent updates to distinct objects are not subject to the one update per second limit.

So how to upload new 20 milions diffrence object google storage ?

P/s : i have ~20M small file in HDD , so i want to upload all to google cloud storage .

1
google-cloud-storage
the question is, are you updating 1 object 20 million times or creating 20 million new objects? – Paul Collingwood
i have ~20M small file in HDD , so i want to upload all to google cloud storage . – Bui Dinh Ngoc
use the command line tool probably then.... – Paul Collingwood

1 Answers

2
votes

I'm sorry, that wording is somewhat misleading. You may upload a particular object of the same object name once per second. If you have a bucket foo and an object bar, you should only upload a new copy of foo/bar about once per second.

You can upload objects with distinct names as fast as your bandwidth permits.