Visualization of Keras Convolution Layer Outputs

Question

I have written the following code for this question where there are two convolution layers (Conv1 and Conv2 for short) and I would like to plot all the outputs of each layer (it's self-contained). Everything is fine for Conv1, but I am missing something about Conv2.

I am feeding a 1x1x25x25 (num images, num channels, height, width (my convention, neither TF or Theano convention)) image to Conv1 which has FOUR 5x5 filters. That means its output shape is 4x1x1x25x25 (num filters, num images, num channels, height, width), resulting in 4 plots.

Now, this output is being fed to Conv1 which has SIX 3x3 filters. Hence, the output of Conv2 should be 6x(4x1x1x25x25), but it is not! It's rather 6x1x1x25x25. That means, there are only 6 plots rather than 6x4, but why? The following functions also prints the shape of each output which they are

(1, 1, 25, 25, 4)
-------------------
(1, 1, 25, 25, 6)
-------------------

but should be

(1, 1, 25, 25, 4)
-------------------
(1, 4, 25, 25, 6)
-------------------

Right?

import numpy as np
#%matplotlib inline #for Jupyter ONLY
import matplotlib.pyplot as plt

from   keras.models     import Sequential
from   keras.layers     import Conv2D
from   keras            import backend as K

model = Sequential()

# Conv1
conv1_filter_size = 5
model.add(Conv2D(nb_filter=4, nb_row=conv1_filter_size, nb_col=conv1_filter_size, 
                 activation='relu', 
                 border_mode='same',
                 input_shape=(25, 25, 1)))

# Conv2
conv2_filter_size = 3
model.add(Conv2D(nb_filter=6, nb_row=conv2_filter_size, nb_col=conv2_filter_size, 
                 activation='relu', 
                 border_mode='same'))

# The image to be sent through the model
img = np.array([
[[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.]],
[[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.]],
[[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.]],
[[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[0.],[1.],[1.],[1.],[1.],[1.]],
[[1.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[1.]],
[[1.],[1.],[1.],[1.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.]],
[[1.],[1.],[1.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[1.],[1.],[1.]],
[[1.],[1.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[0.],[0.],[0.],[1.],[0.],[0.],[0.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[1.],[1.]],
[[1.],[1.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[0.],[0.],[0.],[1.],[0.],[0.],[0.],[0.],[0.],[1.],[1.],[1.],[1.],[0.],[1.],[1.]],
[[1.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[1.]],
[[1.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[1.]],
[[1.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[1.]],
[[1.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[1.]],
[[1.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[1.]],
[[1.],[0.],[0.],[1.],[1.],[1.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[1.],[1.],[1.],[0.],[0.],[1.]],
[[1.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[1.]],
[[1.],[1.],[0.],[1.],[1.],[1.],[1.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[0.],[1.],[1.]],
[[1.],[1.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[1.],[1.]],
[[1.],[1.],[1.],[0.],[0.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[1.],[1.],[1.],[0.],[0.],[1.],[1.],[1.]],
[[1.],[1.],[1.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[1.],[1.],[1.]],
[[1.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[1.]],
[[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[1.],[1.],[1.],[1.],[1.]],
[[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.]],
[[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[0.],[0.],[0.],[0.],[0.],[0.],[0.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.]],
[[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.]]])

def get_layer_outputs(image):
    '''This function extracts the numerical output of each layer.'''
    outputs    = [layer.output for layer in model.layers]
    comp_graph = [K.function([model.input] + [K.learning_phase()], [output]) for output in outputs]

    # Feeding the image
    layer_outputs_list = [op([[image]]) for op in comp_graph]

    layer_outputs = []
    for layer_output in layer_outputs_list:
        print(np.array(layer_output).shape, end='\n-------------------\n')
        layer_outputs.append(layer_output[0][0])

    return layer_outputs

def plot_layer_outputs(image, layer_number):
    '''This function handels plotting of the layers'''
    layer_outputs = get_layer_outputs(image)

    x_max = layer_outputs[layer_number].shape[0]
    y_max = layer_outputs[layer_number].shape[1]
    n     = layer_outputs[layer_number].shape[2]

    L = []
    for i in range(n):
        L.append(np.zeros((x_max, y_max)))

    for i in range(n):
        for x in range(x_max):
            for y in range(y_max):
                L[i][x][y] = layer_outputs[layer_number][x][y][i]


    for img in L:
        plt.figure()
        plt.imshow(img, interpolation='nearest')

plot_layer_outputs(img, 1)

Miladiouss Miladiouss · Accepted Answer · 2017-04-23T01:38:11

The output of a convolution layer is bundled as one image with multiple channels. These could be thought as feature channels, in contrast with color channels. For example, if a convolution layer has F number of filters, it will output an image with F number of channels, no matter how many (color or feature) channels the input image had. This is why Conv2 produces 6 feature maps rather than 6x4. In more details, a convolution filter will convolve over all input channels and the linear combination of its convolution would be fed to its activation function.

Visualization of Keras Convolution Layer Outputs

1 Answers