python - loading saved keras model from gs to pydatalab

Question

My keras model is saved in google storage with model.save(model_name)

I cannot load the model on pydatalab. When I save the model on my local machine, I can just open it with load_model(filepath). Also I did import keras.backend as K, based on NameError when opening Keras model that uses Tensorflow Backend

I have tried the following:

1. model = load_model(tf.gfile.Open(model_file))
   

Error: TypeError: expected str, bytes or os.PathLike object, not GFile

2. load_model('gs://mybucket/model.h5')
   

Error: IOError: Unable to open file (unable to open file: name = 'gs://mybucket/model.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

3. with file_io.FileIO(model_file, 'r') as f:
   modl = load_model(f)
   

error: TypeError: expected str, bytes or os.PathLike object, not FileIO

Answer 1

Load the file from gs storage

from tensorflow.python.lib.io import file_io
model_file = file_io.FileIO('gs://mybucket/model.h5', mode='rb')

Save a temporary copy of the model locally

temp_model_location = './temp_model.h5'
temp_model_file = open(temp_model_location, 'wb')
temp_model_file.write(model_file.read())
temp_model_file.close()
model_file.close()

Load model saved locally

model = load_model(temp_model_location)

Answer 2

I don't think Keras supports the TensorFlow file system which in turn knows how to read from GCS.

You could try downloading from GCS to a local path, and then reading from that to load the model.

Answer 3

The following function works for retraining an already trained keras model (with new data) on gcloud machine learning platform (Thanks to Tíarnán McGrath).

def load_models(model_file):

    model = conv2d_model() #the architecture of my model, not compiled yet
    file_stream = file_io.FileIO(model_file, mode='r')
    temp_model_location = './temp_model.h5'
    temp_model_file = open(temp_model_location, 'wb')
    temp_model_file.write(file_stream.read())
    temp_model_file.close()
    file_stream.close()
    model.load_weights(temp_model_location)

    return model

For some reason, load_model from keras.models does not work for me anymore, so I have to build the model each time.

Answer 4

OS level command can also be used just in case someone is using Colab

To mound your google drive use

from google.colab import drive
drive.mount('/content/drive', force_remount=True)

Code to mount GCS

from google.colab import auth
auth.authenticate_user()
project_id = 'thirumalai_bucket'  #your bucket here
!gcloud config set project {project_id}
!gsutil ls

!gsutil -m cp

in your case:

!gsutil -m cp gs://mybucket/model.h5  /content/drive/My\ Drive/models/ 

now the file model.h5 available in drive /content/drive/My Drive/models/ move to your models directory using:

!cd /content/drive/My\ Drive/models/

load_model('model.h5')

Hope this helps!