How to improve slow requests per second rate when downloading from google cloud storage bucket

Question

I'm fetching data from my google cloud bucket with @google-cloud/storage library. However I'm not able to get more than ~5 downloads/second from the bucket.

const Storage = require('@google-cloud/storage');
const storage = Storage({ keyFilename: './gcloud-api-creds.json' });
const bucket = storage.bucket('my-bucket');

Promise.all(Array.from(Array(80)).map(
  (d,i) => bucket.file(`index.html`)
    .download()
    .then(() => console.log(`done ${i}`))
)).then(() => console.log("READY"));

Takes around ~14 seconds to complete 80 download requests. I believe I'm hitting some per user limit of storage.

Google Cloud Storage docs claims supporting ~5000 req/s by default

There is no limit to reads of an object. Buckets initially support roughly 5000 reads per second and then scale as needed. (https://cloud.google.com/storage/quotas)

How can I achieve that rate?

Answer 1

After discussing with google cloud support team we found out that it was actually used bandwidth that was limiting the amount of request / second on the app engine flex container.

Looks like there was only 65mbit download bandwidth between instance and cloud storage bucket according to gsutil perf test.

Answer 2

I think that the issue is not the @google-cloud/storage library or any rate limit, but how the map method is used.

Array.map is executed synchronously, therefore if you wait each time to finish a download before starting a new one you perform each request sequentially even if you are using the Promise.all and not in parallel since you are no creating any promise when working on the array. Therefore you go slower than expected.

I think that you might find this example really useful {source}:

var arr = [1, 2, 3, 4, 5];

var results: number[] = await Promise.all(arr.map(async (item): Promise<number> => {
    await callAsynchronousOperation(item);
    return item + 1;
}));

According to the MDN docs for Promise.all:

The Promise.all(iterable) method returns a promise that resolves when all of the promises in the iterable argument have resolved, or rejects with the reason of the first passed promise that rejects.