Tensorflow: Summaries defined in function not accessible in tensorboard

Question

I have a graph and a set of custom functions that define multilayer RNNs according to an input list which will specify the number of units in each layer. For instance:

def BuildLayers(....):
    # takes inputs, list of layer sizes, mask information, etc
    #
    # invokes BuildLayer(...) several times
    #
    # returns RNN output and states of last layer

BuildLayer loops through a more detailed function which builds and returns individual layers:

def BuildLayer(....):
    # Takes individual layer size, output of previous layer, etc
    #
    # handles bookkeeping of RNNCells, wrappers, reshaping, etc
    # **Important!  Defines scope for each layer**
    #
    # returns RNN output and states of last layer

And ultimately this would be called in a function that defines a graph and runs it in a session:

def Experiment(parameters):
    tf.reset_default_graph()
    graph = tf.Graph()
    with graph.as_default():    
        #
        # Placeholders
        # BuildLayers(...)
        # Loss function definitions
        # optimizer definitions
    with tf.Session(graph=graph) as session:
        #
        # Loop through epochs:
            # etc

I.e., if the layer size parameter is [16, 32, 16], we end up with an RNN that has a cell of 16 units in layer1, scoped as layer1, 32 units in layer 2, scoped appropriately, and 16 units in layer 3, scoped, etc.

This seems to work fine, a casual inspection of the graph in tensorboard looks correct, nodes look correct, the thing trains, etc.

Problem: How can I add histogram summaries, e.g., of kernel weights and biases, to that function definition? I've done so naively, as such:

def buildLayer(numUnits, numLayer, input, lengths):
    name = 'layer' "{0:0=2d}".format(numLayer)
    with tf.variable_scope(name):
        cellfw = tf.contrib.rnn.GRUCell(numUnits, activation = tf.nn.tanh)       
        cellbw = tf.contrib.rnn.GRUCell(numUnits, activation = tf.nn.tanh)           
        outputs,  state  = tf.nn.bidirectional_dynamic_rnn(cell_fw = cellfw, cell_bw = cellbw, inputs = input, dtype=tf.float32, sequence_length = lengths)
        outputs             = tf.concat([outputs[0], outputs[1]], axis=2)

        FwKernel     = tf.get_default_graph().get_tensor_by_name(name + '/bidirectional_rnn/fw/gru_cell/gates/kernel:0')
        FwKernel_sum = tf.summary.histogram("FwKernel", FwKernel, 'rnn')
        return outputs, state

And then, at the end of the graph definition, assumed these summaries would be caught up in the

merged = tf.summary.merge_all()

statement. It isn't. I'm confused by this behavior. I can see the histogram summary definitions on a visual inspection of the graph in tensorboard-- they're there. But they don't seem to be getting to the merge and so are never accessible in tensorboard as histograms per se.

How do I get summaries, which are defined in a function, to show up in tensorboard, preferably through a merge and without passing them around through function calls like excess baggage?

Novak Novak · Accepted Answer · 2018-04-16T22:04:25

The least painful way I have found to avoid this is to pass a single list (i.e., "summaries") through each function, and within the BuildLayers function, to append or extend that list with all desired histogram summaries.

Then, in the main graph definition, rather than a merge_all

merged = tf.summary.merge_all()

instead use a merge and pass the list in as the argument

merged = tf.summary.merge(summaries)

This has the disadvantage of not actually being a merge_all, meaning that if you had defined other summaries (typically scalar summaries for loss functions, at least) you're going to have to manually append them to the summaries list or carry around two merge objects or something similar, which misses the self-advertised point of the merge_all.

I leave this here as an answer to my own question because it might help someone, but will pointedly not accept it because I am hoping to be shown a better way.

Tensorflow: Summaries defined in function not accessible in tensorboard

2 Answers