List only files from the Google cloud storage

Question

In GCS I have bucket XYZ, under that I have folder JM, under that I have files. For example:

XYZ/JM/file1.tar.gz,XYZ/JM/file2.tar.gz,XYZ/JM/file3.tar.gz,XYZ/JM/file4.tar.gz etc.

Using the code below I am able to list the files but its displaying the full path like:

JM/file1.tar.gz,JM/file2.tar.gz,JM/file3.tar.gz

Code:

from google.cloud import storage
storage_client = storage.Client.from_service_account_json()

BucketName="XYZ"
bucket=storage_client.get_bucket(BucketName)


filename=list(bucket.list_blobs(prefix="jm/"))
for name in filename:
       print(name.name)

Query: I want to list the files under folder JM. I don't want to display JM in the list, just display file ex: file1.tar.gz,file2.tar.gz

Daniel Ocando Daniel Ocando · Accepted Answer · 2020-07-07T10:01:32

Everything in Cloud Storage is considered an object (even folders). Notice that as stated on the documentation:

To the service, the object gs://your-bucket/abc/def.txt is just an object that happens to have "/" characters in its name. There is no "abc" directory; just a single object with the given name.

and that is the reason why you receive the full object "path" which is actually the object's real name when using the list_blobs() method.

The prefix parameter of the list_blobs() method function you are using to filter the blobs should suffice to list the specific objects that you are looking for.

But afterwards you'd need to consider using a regex or a similar string splitting method by splitting with the '/' character to get just the portion of the blob's name that you consider relevant.

EDIT

I tested the following and it worked:

from google.cloud import storage
storage_client = storage.Client.from_service_account_json()

BucketName="XYZ"
bucket=storage_client.get_bucket(BucketName)


filename=list(bucket.list_blobs(prefix="jm/"))
for name in filename:
    try:
        prefix, object_name = name.name.split('/')
    except:
        print("An error occurred splitting the string.")
    print(object_name)

List only files from the Google cloud storage

1 Answers