Visualizing output of convolutional layer in tensorflow

Question

I'm trying to visualize the output of a convolutional layer in tensorflow using the function tf.image_summary. I'm already using it successfully in other instances (e. g. visualizing the input image), but have some difficulties reshaping the output here correctly. I have the following conv layer:

img_size = 256
x_image = tf.reshape(x, [-1,img_size, img_size,1], "sketch_image")

W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])

h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)

So the output of h_conv1 would have the shape [-1, img_size, img_size, 32]. Just using tf.image_summary("first_conv", tf.reshape(h_conv1, [-1, img_size, img_size, 1])) Doesn't account for the 32 different kernels, so I'm basically slicing through different feature maps here.

How can I reshape them correctly? Or is there another helper function I could use for including this output in the summary?

Answer 1

I don't know of a helper function but if you want to see all the filters you can pack them into one image with some fancy uses of tf.transpose.

So if you have a tensor that's images x ix x iy x channels

>>> V = tf.Variable()
>>> print V.get_shape()

TensorShape([Dimension(-1), Dimension(256), Dimension(256), Dimension(32)])

So in this example ix = 256, iy=256, channels=32

first slice off 1 image, and remove the image dimension

V = tf.slice(V,(0,0,0,0),(1,-1,-1,-1)) #V[0,...]
V = tf.reshape(V,(iy,ix,channels))

Next add a couple of pixels of zero padding around the image

ix += 4
iy += 4
V = tf.image.resize_image_with_crop_or_pad(image, iy, ix)

Then reshape so that instead of 32 channels you have 4x8 channels, lets call them cy=4 and cx=8.

V = tf.reshape(V,(iy,ix,cy,cx)) 

Now the tricky part. tf seems to return results in C-order, numpy's default.

The current order, if flattened, would list all the channels for the first pixel (iterating over cx and cy), before listing the channels of the second pixel (incrementing ix). Going across the rows of pixels (ix) before incrementing to the next row (iy).

We want the order that would lay out the images in a grid. So you go across a row of an image (ix), before stepping along the row of channels (cx), when you hit the end of the row of channels you step to the next row in the image (iy) and when you run out or rows in the image you increment to the next row of channels (cy). so:

V = tf.transpose(V,(2,0,3,1)) #cy,iy,cx,ix

Personally I prefer np.einsum for fancy transposes, for readability, but it's not in tf yet.

newtensor = np.einsum('yxYX->YyXx',oldtensor)

anyway, now that the pixels are in the right order, we can safely flatten it into a 2d tensor:

# image_summary needs 4d input
V = tf.reshape(V,(1,cy*iy,cx*ix,1))

try tf.image_summary on that, you should get a grid of little images.

Below is an image of what one gets after following all the steps here.

Answer 2

In case someone would like to "jump" to numpy and visualize "there" here is an example how to display both Weights and processing result. All transformations are based on prev answer by mdaoust.

# to visualize 1st conv layer Weights
vv1 = sess.run(W_conv1)

# to visualize 1st conv layer output
vv2 = sess.run(h_conv1,feed_dict = {img_ph:x, keep_prob: 1.0})
vv2 = vv2[0,:,:,:]   # in case of bunch out - slice first img


def vis_conv(v,ix,iy,ch,cy,cx, p = 0) :
    v = np.reshape(v,(iy,ix,ch))
    ix += 2
    iy += 2
    npad = ((1,1), (1,1), (0,0))
    v = np.pad(v, pad_width=npad, mode='constant', constant_values=p)
    v = np.reshape(v,(iy,ix,cy,cx)) 
    v = np.transpose(v,(2,0,3,1)) #cy,iy,cx,ix
    v = np.reshape(v,(cy*iy,cx*ix))
    return v

# W_conv1 - weights
ix = 5  # data size
iy = 5
ch = 32   
cy = 4   # grid from channels:  32 = 4x8
cx = 8
v  = vis_conv(vv1,ix,iy,ch,cy,cx)
plt.figure(figsize = (8,8))
plt.imshow(v,cmap="Greys_r",interpolation='nearest')

#  h_conv1 - processed image
ix = 30  # data size
iy = 30
v  = vis_conv(vv2,ix,iy,ch,cy,cx)
plt.figure(figsize = (8,8))
plt.imshow(v,cmap="Greys_r",interpolation='nearest')

Answer 3

you may try to get convolution layer activation image this way:

    h_conv1_features = tf.unpack(h_conv1, axis=3)
    h_conv1_imgs = tf.expand_dims(tf.concat(1, h_conv1_features_padded), -1)

this gets one vertical stripe with all images concatenated vertically.

if you want them padded (in my case of relu activations to pad with white line):

    h_conv1_features = tf.unpack(h_conv1, axis=3)
    h_conv1_max = tf.reduce_max(h_conv1)
    h_conv1_features_padded = map(lambda t: tf.pad(t-h_conv1_max, [[0,0],[0,1],[0,0]])+h_conv1_max, h_conv1_features)
    h_conv1_imgs = tf.expand_dims(tf.concat(1, h_conv1_features_padded), -1)

Answer 4

I personally try to tile every 2d-filter in a single image.

For doing this _{-if i'm not terribly mistaken since I'm quite new to DL-} I found out that it could be helpful to exploit the depth_to_space function, since it takes a 4d tensor

[batch, height, width, depth]

and produces an output of shape

[batch, height*block_size, width*block_size, depth/(block_size*block_size)]

Where block_size is the number of "tiles" in the output image. The only limitation to this is that the depth should be the square of block_size, which is an integer, otherwise it cannot "fill" the resulting image correctly. A possible solution could be of padding the depth of the input tensor up to a depth that is accepted by the method, but I sill havn't tried this.

Answer 5

Another way, which I think very easy, is using the get_operation_by_name function. I had hard time visualizing the layers with other methods but this helped me.

#first, find out the operations, many of those are micro-operations such as add etc.
graph = tf.get_default_graph()
graph.get_operations()

#choose relevant operations
op_name = '...' 
op = graph.get_operation_by_name(op_name)
out = sess.run([op.outputs[0]], feed_dict={x: img_batch, is_training: False}) 
#img_batch is a single image whose dimensions are (1,n,n,1). 

# out is the output of the layer, do whatever you want with the output
#in my case, I wanted to see the output of a convolution layer
out2 = np.array(out)
print(out2.shape)

# determine, row, col, and fig size etc.
for each_depth in range(out2.shape[4]):
    fig.add_subplot(rows, cols, each_depth+1)
    plt.imshow(out2[0,0,:,:,each_depth], cmap='gray')

For example below is the input(colored cat) and output of the second conv layer in my model.

Note that I am aware this question is old and there are easier methods with Keras but for people who use an old model from other people (such as me), this may be useful.