TensorBoard recording stats separate from training

Question

I'm trying to use TensorBoard to display some graphs of a neural network training run. (That is, graphs of test and validation accuracy during training, not just of the network structure.) There is some example code

As well as some questions on this site, all of which seem to follow the same pattern as the example code. That is, the pattern always revolves around something like

summary, _ = sess.run([merged, train_step], ...

So basically, the operations of running a training step and recording statistics for graph display, are being conflated.

This is fine as far as it goes, but I'm trying to retrofit the graph to an existing program that inevitably does things in a slightly different way, so the example code won't work as is. What I really want to do is isolate some code that just records the statistics, separate from existing code to do the training.

How do you record statistics for TensorBoard, within the main training loop, but separate from the code that does the training?

To put it differently, at each training step you run some code that eventually returns some scalar value (call it accuracy) and you want to write it into tensorboard summary using separate sess.run(...), did I understand your question correctly? — abhuse
@openmark At each training step I run some code that eventually returns a scalar value, and I want to write it into tensorboard summary in a separate line of code, yes. Is there a reason writing tensorboard summary needs to involve sess.run? If so, then yes, in a separate call to that. — rwallace

abhuse abhuse · Accepted Answer · 2018-06-12T23:21:34

You can manually create tf.Summary object that stores the scalar value and pass it to tf.summary.FileWriter like in the following example:

summary_writer = tf.summary.FileWriter("path_to_log_dir")
# ...
for i in range(max_training_steps):
  # compute the values of interest
  scalar_value_1 = ...
  # ...
  scalar_value_n = ...

  # manually create tf.Summary object
  summary = tf.Summary(
    value=[tf.Summary.Value(tag="Metrics_1", simple_value=scalar_value_1),
           # ...
           tf.Summary.Value(tag="Metrics_n", simple_value=scalar_value_n)])
  summary_writer.add_summary(summary, i)
# ...
summary_writer.close()

Alternatively, you can define tf.summary.scalar() operation using tf.placeholder as a tensor and feed the actual value at run time:

scalar_pl_1 = tf.placeholder(tf.float32)
tf.summary.scalar("Metrics_1", scalar_pl_1)
# ...
scalar_pl_n = tf.placeholder(tf.float32)
tf.summary.scalar("Metrics_n", scalar_pl_n)

# Merge all summaries
merged = tf.summary.merge_all()

summary_writer = tf.summary.FileWriter("path_to_log_dir")

with tf.Session() as sess:
  for i in range(max_training_steps):
    # compute scalar values of interest
    scalar_value_1 = ...
    scalar_value_n = ...

    feed_dict = {scalar_pl_1: scalar_value_1, scalar_pl_n: scalar_value_n}
    summary = sess.run(merged, feed_dict=feed_dict)
    summary_writer.add_summary(summary, i)
# ...
summary_writer.close()

TensorBoard recording stats separate from training

1 Answers