Load model from Google Cloud Storage without downloading

Question

Is there a way to serve model from Google Cloud Storage without actually downloading a copy of model? like streaming the data directly?

I'm trying to load a fasttext model that is hosted on Google Cloud Storage. everytime i run the program, it needs to get and download a copy of that model in the bucket.

language_model_filename = 'lid.176.bin' // filename in GCS
language_model_local = 'lid.176.bin' // local file name when downloaded
bucket = storage_client.get_bucket(CLOUD_STORAGE_BUCKET)
blob = bucket.blob(language_model_filename)
blob.download_to_filename(language_model_local)
language_model = FastText.load_model(language_model_local)

Pablo Almécija Rodríguez Pablo Almécija Rodríguez · Accepted Answer · 2019-03-15T10:00:07

You can use Streaming Tranfers for that purpose. As explained in the documentation, you can use the third party boto client library plugin for Cloud Storage.

A streaming download example would look like this:

import sys

downloaded_file = 'saved_data_file'
MY_BUCKET = 'my_app_bucket'
object_name = 'data_file'
src_uri = boto.storage_uri(MY_BUCKET + '/' + object_name, 'gs')
src_uri.get_key().get_file(sys.stdout)

Load model from Google Cloud Storage without downloading

1 Answers