How to get train loss and evaluate loss every global step in Tensorflow Estimator?

Question

I can get traing loss every global step. But I do want to add the evaluate loss in graph 'lossxx' in tensorboard. How to do that?

  class MyHook(tf.train.SessionRunHook):
    def after_run(self,run_context,run_value):
      _session = run_context.session
      _session.run(_session.graph.get_operation_by_name('acc_op'))

  def my_model(features, labels, mode):
    ...
    logits = tf.layers.dense(net, 3, activation=None)
    predicted_classes = tf.argmax(logits, 1)
    if mode == tf.estimator.ModeKeys.PREDICT:
      predictions = {
        'class': predicted_classes,
        'prob': tf.nn.softmax(logits)
      }
      return tf.estimator.EstimatorSpec(mode, predictions=predictions)

    # Compute loss.
    loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)
    acc, acc_op = tf.metrics.accuracy(labels=labels,   predictions=predicted_classes)
    tf.identity(acc_op,'acc_op')
    loss_sum = tf.summary.scalar('lossxx',loss)
    accuracy_sum = tf.summary.scalar('accuracyxx',acc)
    merg = tf.summary.merge_all()

    # Create training op.
    if mode == tf.estimator.ModeKeys.TRAIN:
      optimizer = tf.train.AdagradOptimizer(learning_rate=0.1)
      train_op = optimizer.minimize(loss,  global_step=tf.train.get_global_step())
      return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=train_op,
                                      training_chief_hooks=[
                                            tf.train.SummarySaverHook(save_steps=10, output_dir='./model', summary_op=merg)])

    return tf.estimator.EstimatorSpec(
        mode, loss=loss, eval_metric_ops={'accuracy': (acc, acc_op)}
    )


  classifier.train(input_fn=train_input_fn, steps=1000,hooks=[ MyHook()])

The estimator train function normally only uses training data as input. You would need to have some validation data in each batch? Or alternate training batches and validation batches? — Olivier Moindrot
just to make sure I understood you correctly: you want to overlay the loss of the evaluation with the loss during training? If so, do you want the loss to be averaged over all samples in the validation set? Otherwise, how do you want to merge the x-axis (global_step) of lossxx with the x-axis from the evaluation? — maddin25

maddin25 maddin25 · Accepted Answer · 2018-07-13T12:33:15

You actually don't need to create a SummarySaverHook by yourself, as it is already included in the tf.estimator.Estimator. Just create all the summaries you want with tf.summary.xxx and they will all be evaluated every n steps. (See tf.estimator.RunConfig for this).

Also, you don't need to create a summary for your final loss loss. This will also be created for you automatically. If you do it like this, then the training and evaluation summaries will be shown in the same graph on TensorBoard. The estimator creates a sub-directory eval in your current model_dir to achieve this.

And a small hint: use the acc_op directly in summaries to update the metric and get the value of it. However, the tf.metrics functions are quite difficult to handle ;-)

How to get train loss and evaluate loss every global step in Tensorflow Estimator?

2 Answers