Google Cloud Storage NodeJS and JSON API not returning certain folders when trying to list

Question

While trying to use the NodeJS Client Library (getFiles - https://googleapis.dev/nodejs/storage/latest/Bucket.html#getFiles) and the JSON API (objects/list - https://cloud.google.com/storage/docs/json_api/v1/objects/list) The response seems to not return certain folders in both cases.

While using gsutil ls it lists all the folders as expected and using the Google Cloud Console to browse the GCS bucket also lists all files as expected.

Has there been change in the Storage API that could be causing this to happen?

UPDATE:

The code to list GCS Folder paths, that uses the NodeJS Client SDK for Storage:

import * as functions from 'firebase-functions';
import { Storage } from '@google-cloud/storage';
import { globVars } from '../admin/admin';

const projectId = process.env.GCLOUD_PROJECT;
// shared global variables setup
const { keyFilename } = globVars;
// Storage set up
const storage = new Storage({
  projectId,
  keyFilename,
});

export const gcsListPath = functions
  .region('europe-west2')
  .runWith({ timeoutSeconds: 540, memory: '256MB' })
  .https.onCall(async (data, context) => {
    if (context.auth?.token.email_verified) {
      const { bucketName, prefix, pathList = false, fileList = false } = data;
      let list;
      const options = {
        autoPaginate: false,
        delimiter: '',
        prefix,
      };

      if (pathList) {
        options.delimiter = '/';
        const verboseResponse = await storage
          .bucket(bucketName)
          .getFiles(options);
        list = verboseResponse[2].prefixes;
      }
      if (fileList) {
        const [files] = await storage
          .bucket(bucketName)
          .getFiles(options);
        list = files.map((file) => file.name);
      }

      return { list };

    } else {
      return {
        error: { message: 'Bad Request', status: 'INVALID_ARGUMENT' },
      };
    }
  });

Can you share the code that you wrote? To be able to reproduce that on our side — guillaume blaquiere
@guillaumeblaquiere, thanks for your message. I've updated the question with the code we use to list GCS folders. The odd thing is that we're seeing the same issue when we use the JSON_API. — Trevor Lazarus
One more thing to note is that this issue is only recent and we've been using the API to list folder paths with no problems for the past 3 months. — Trevor Lazarus

Trevor Lazarus Trevor Lazarus · Accepted Answer · 2021-05-05T12:42:30

Looks like the pagination feature was limiting the results. Implementing nextQuery returns the next page of listings.

Google Cloud Storage NodeJS and JSON API not returning certain folders when trying to list

1 Answers