Tensorflow/Cloud Datalab : Restore called with invalid save path

0
votes

I am training a model in the Cloud with Cloud ML from a Cloud Datalab notebook. I'd like to evaluate the model locally with.

%mlalpha train
package_uris: $package_path_local
python_module: trainer.task
args:
  eval_data_paths :  ...# local or gs://

I all cases (local or remote I get the following error:

Restore called with invalid save path: u'[path to checkpoint file]'. File path is: u'[path to checkpoint file]'

However evaluating on the cloud works fine. Tensorflow version in latest Cloud Datalab is 0.11rc0

2
tensorflowgoogle-cloud-datalabgoogle-cloud-ml

2 Answers

1
votes

Possible problems:

  1. your training model graph and evaluation model graph are different. Check to make sure you are using the same model.

  2. You trained a model, then changed the model, and tried to train or evaluate again. You may need to delete the checkpoint directory so that things start afresh.

0
votes

You were running "%mlalpha train" command, so it is training but not evaluation? Or does your code in training package try to load a checkpoint?

When the evaluation worked in cloud, what command did you run?