Google Cloud ML Engine does not return objective values when hyperparameter tuning

Question

In the training output for a hyperparameter tuning job on Google Cloud ML Engine, I do not see the values of the objective calculated for each trial. The training output is the following:

    {
  "completedTrialCount": "4",
  "trials": [
    {
      "trialId": "2",
      "hyperparameters": {
        "learning-rate": "0.0010000350944297609"
      }
    },
    {
      "trialId": "3",
      "hyperparameters": {
        "learning-rate": "0.0053937227881987841"
      }
    },
    {
      "trialId": "4",
      "hyperparameters": {
        "learning-rate": "0.099948384760813816"
      }
    },
    {
      "trialId": "1",
      "hyperparameters": {
        "learning-rate": "0.02917661111653325"
      }
    }
  ],
  "consumedMLUnits": 0.38,
  "isHyperparameterTuningJob": true
}

The hyperparameter tuning job appears to run correctly and displays a green check mark next to the job. However, I expected that I would see the value of the objective function for each trial in the training output. Without this, I don't know which trial is best. I have attempted to add the value of the objective into the summary graph as follows:

with tf.Session() as sess:
    ...
    final_cost = sess.run(tf.reduce_sum(tf.square(Y-y_model)), feed_dict={X: trX, Y:trY})
    summary = Summary(value=[Summary.Value(tag='hyperparameterMetricTag', simple_value=final_cost)])
    summary_writer.add_summary(summary)
    summary_writer.flush()

I believe I have followed all the steps discussed in the documentation to set up a hyperparameter tuning job. What else is required to ensure that I get an output that lets me compare different trials?

What is the path you've configured for summary_writer? Is there any chance that the actually value is 0.0? — rhaertel80
I configured the path this way: eval_path = os.path.join(args.jobDir, 'metric1') summary_writer = tf.summary.FileWriter(eval_path). args.jobDir is the GCS bucket where the job files get stored. When I run the code locally, the final_cost value is non-zero. — Dan
Could you please check if you can find the value of hyperparameterMetricTag on tensorboard. And please make sure you specify the same hyperparameterMetricTag name(it's hyperparameterMetricTag in your case) in your job request(HyperparameterSpec) and your code. — lwz1992
Thanks @lwz1992. I think the problem was a mismatch between the hyperparameterMetricTag value in the configuration file and the tag used in the Summary() in the code. I set them equal and now I see the objectiveValue in the training output. — Dan

lwz1992 lwz1992 · Accepted Answer · 2018-05-04T16:22:35

Could you please check if you can find the value of hyperparameterMetricTag on tensorboard to make sure you report the metric correctly? And please make sure you specify the same hyperparameterMetricTag name(it's hyperparameterMetricTag in your case) in your job request(HyperparameterSpec) and your code.

Google Cloud ML Engine does not return objective values when hyperparameter tuning

1 Answers