Access Spacy's trained model/folder in google cloud storage

Question

I was deploying a python application which uses Spacy's custom trained model on GAE. I have also uploaded model folder in Google Cloud Storage but am facing issues in getting only folder from storage bucket and using that to load in spacy.

Right now, if I am trying to get a blob, I can get only one file, not a folder.

Please help me.

FYI, Spacy's custom trained and saved model is a folder that contains multiple files. Thanks

EDIT 1 :

This is the directory structure of saved model

Chris32 Chris32 · Accepted Answer · 2020-05-28T07:38:01

Google Cloud Storage doesn't have folders in the reality, what you see as folders are just a representation, you can see a more detailed explanation here.

What you have to do is to fetch all the files inside of a "folder" recursively. i.e.:

import sys
from google.cloud import storage
from google.cloud.storage.blob import Blob


client = storage.Client()
for blob in client.list_blobs('mybucket', prefix='sofolder'):
    blobname = blob.name
    blobstring = str(blobname)
    blobcleaname = blobstring.rsplit('/', 1)[-1]
    if not blobstring.endswith('/'):
        blob.download_to_filename('./' + str(blobcleaname))
        print(blobstring)

Update 1:

So I did a quick reproduction of your use case by just printing to console the name of my bucket. I have a structure similar to yours:

bucketname
    -filexxx 
    -folderyyy
    -sofolder <--- the folder i'm interested in
      -file1.png
      -folder_a
        -fileinfolder_a.png
      -folder_b
        -fileinfolder_b.png
      -folder_c
        -fileinfolder_c.png

and by runnning this:

import sys
from google.cloud import storage
from google.cloud.storage.blob import Blob

client = storage.Client()
for blob in client.list_blobs('bucketname', prefix='sofolder'):
    blobname = blob.name
    blobstring = str(blobname)
    if not blobstring.endswith('/'):
        print(blobstring)

I'm getting this output:

sofolder/
sofolder/file1.png
sofolder/folder_a/fileinfolder_a.png
sofolder/folder_b/fileinfolder_b.png
sofolder/folder_c/fileinfolder_c.png

Access Spacy's trained model/folder in google cloud storage

1 Answers