Retrieve and download all files from a set of paths in Google Cloud Storage using NodeJS

Question

I have an array entryIdsWithImages that are like folder names in Google cloud storage from where I have to download all the images, i.e., I want to download all the images from these specific GCP paths. So, according to the @google-cloud/storage docs, I first retrieve all the files from these paths, then downloaded them into a temporary directory, which I later zip. However, I found that the images downloaded were having a size of 0B, whereas they're not in the actual storage. Where am I wrong?

await Promise.all(
    entryIdsWithImages.map(async (entryId) => {
        const prefix = `images/${uid}/${entryId}`;
        // download the images for the entry.
        const [files] = await bucket.getFiles({
        prefix,
        });
        files.forEach(async (file) => {
        // file.name includes the whole path to the file, thus extract only the innermost path, which will be the name of the file
        const fileName = file.name.slice(
            file.name.lastIndexOf('/') + 1
        );
        const imgTempFilePath = path.join(imageDirectory, fileName);
        try {
            await file.download({
            destination: `${imgTempFilePath}.jpg`,
            });
        } catch (e) {
            console.log(
            `Error downloading the image at ${prefix}/${fileName}: `,
            e
            );
        }
        });
    })
)

The version I'm using is: "@google-cloud/storage": "^4.7.0". I tried with the latest version, only to obtain the same results.

What happens if you download a single file with a known filename? await storage.bucket(bucketName).file(srcFilename).download(destination: "foo.jpg"});? — user835611
I have tried your same exact code and it worked with me (extensions and without extensions). I have also tried with 4.7.0 and the latest, same it worked. I suggest that you try: 1- same function but with just one prefix 2- different bucket — Methkal Khalawi

user835611 user835611 · Accepted Answer · 2020-12-13T10:37:55

The Cloud Storage API code looks correct to me. I'm expecting that this is your problem:

destination: `${imgTempFilePath}.jpg`,

Changing the file type to jpg might result in corrupt files that look like 0K.

This code works perfectly find for my bucket:

const downloadAll = async (bucket) => {
    const imageDirectory = 'downloads';
    const prefix = `READ`;
    // download the images for the entry.
    const [files] = await bucket.getFiles({
        prefix,
    });
    files.forEach(async (file) => {
        console.log(`file: ${file}`);
        // file.name includes the whole path to the file, thus extract only the innermost path, which will be the name of the file
        const fileName = file.name.slice(
            file.name.lastIndexOf('/') + 1
        );
        const imgTempFilePath = path.join(imageDirectory, fileName);
        try {
            await file.download({
                destination: `${imgTempFilePath}.jpg`,
            });
        } catch (e) {
            console.log(
                `Error downloading the image at ${prefix}/${fileName}: `,
                e
            );
        }
    });
}

Retrieve and download all files from a set of paths in Google Cloud Storage using NodeJS

2 Answers