Unable to make predictions on google cloud ml, whereas same model is working on the local machine

1
votes

I am trying to train a machine learning model usinf tensorflow library in the google cloud. I am able to train the model in the cloud after creating a bucket. I am facing the issue when I am tring to make predictions using the existing model. The code and the data is available in the following Github directory. https://github.com/terminator172/game-price-predictions

The tensorflow version on the cloud is 1.8 and the tensorflow version on my system is also 1.8

I tried to make predictions by giving the following input "gcloud ml-engine predict --model=earnings --version=v8 --json-instances=sample_input_prescaled.json"

It errored out with the following error "{ "error": "Prediction failed: Error during model execution: AbortionError(code=StatusCode.FAILED_PRECONDITION, details=\"Attempting to use uninitialized value output/biases4\n\t [[Node: output/biases4/read = IdentityT=DT_FLOAT, _output_shapes=[[1]], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]]\")" }"

2
pythontensorflowmachine-learninggoogle-cloud-ml

2 Answers

0
votes

The error message indicates that not all variables have been initialized. There is some sample code in the CloudML samples that demonstrate how to take care of initialization (link) Also, I recommend using tf.saved_model.simple_save on newer versions of TF. Try the following changes to your code:

def main_op():
  init_local = variables.local_variables_initializer()
  init_tables = lookup_ops.tables_initializer()
  return control_flow_ops.group(init_local, init_tables)

[...snip...]    

# This replaces everything from your SavedModelBuilder on
tf.saved_model.simple_save(
    session,
    export_dir='exported_model',
    inputs={'input': X},
    outputs={'earnings': prediction},
    legacy_init_op=main_op)  # This line is important
0
votes

Your model directory in gcloud (the one that you provide with the --model flag) should contain 2 things:

  1. The saved_model.pb file, containing the actual TensorFlow program, or model, and a set of named signatures, each identifying a function that accepts tensor inputs and produces tensor outputs.

  2. The variables directory, containing a standard training checkpoint.

In case your variables directory is missing and you have only the saved_model.pb file, you can get this Attempting to use uninitialized value error. In order to fix it you just need to add the variables directory to your model directory in gcloud.

Reference: Tensorflow SavedModel format