Google Cloud ML FAILED_PRECONDITION

Question

I am trying to use Google Cloud ML to host a Tensorflow model and get predictions. I have a pretrained model that I have uploaded to the cloud and I have created a model and version in my Cloud ML console.

I followed the instructions from here to prepare my data for requesting online predictions. For both the Python method and the glcoud method I get the same error. For simplicity, I'll post the gcloud method:

I run gcloud ml-engine predict --model spell_correction --json-instances test.json where test.json is my input data file (a JSON array named instances). I get the following result:

ERROR: (gcloud.ml-engine.predict) HTTP request failed. Response: {
  "error": {
  "code": 400,
  "message": "Precondition check failed.",
  "status": "FAILED_PRECONDITION"
  }
}

How can I get more details about this? The same exact error happens when I try via Python and there I have a googleapiclient.http.HttpRequest object containing the error. I just want to know why this error is happening other than this generic error. Does anyone know how to get more details via either the Python method or the gcloud method? I am assuming that since it is the same error, it is the same root cause.

Output of gcloud ml-engine models list:

NAME              DEFAULT_VERSION_NAME
spell_correction  testing

Output of gcloud ml-engine versions list --model spell_correction

NAME     DEPLOYMENT_URI
testing  gs://<my-bucket>/output/1/

test.json: {"instances": [{"tokens": [[9], [4], [11], [9]], "mask": [[18], [7], [12], [30]], "keep_prob": 1.0, "beam": 64}]}

My inputs to the model:

tokens: tf.placeholder(tf.int32, shape=[None, None])

mask: tf.placeholder(tf.int32, shape=[None, None])

keep_prob: tf.placeholder(tf.float32)

beam: tf.placeholder(tf.int32)

When calling via python, the request_body is just test.json as a string.

Can you run gcloud ml-engine models list as well as gcloud ml-engine versions list --model spell_correction to verify the model was successfully created? — rhaertel80
Can you post what is being sent in test.json and what is being sent over as request through python? — Bhupesh
I sent an email. Would changing the shape break my checkpoints though? I already had to start over with the upgrade to 1.0, I would like to avoid doing that again if possible — jbird

roger roger · Accepted Answer · 2017-03-14T01:43:26

A side note: did you try "local predict" (https://cloud.google.com/sdk/gcloud/reference/ml-engine/local/predict) with your model first? You might be able to get more information there first.

Google Cloud ML FAILED_PRECONDITION

2 Answers