How keras manages weights of convolutional layers?

Question

I am working with the MNIST data set and I am using keras to train a convolutional neural network. There is something about the weight matrices that I do not understand.

The input layer has 28x28=784 neurons. Then I use:

Conv2D(32,kernel_size=(7,7),stride=(3,3),use_bias=False)
Conv2D(64,kernel_size=(5,5),stride=(2,2),use_bias=False)
Flatten()
Dense(200,use_bias=False)
Dense(150,use_bias=False)
Dense(10,use_bias=False,softmax)

After I train the model and put W = model.get_weights(), I print W[i].shape for each i and obtain:

(7,7,1,32)
(5,5,32,64)
(256,200)
(200,150)
(150,10)

As far as I understand, this means that for the first hidden layer there are 32 images of 8x8=64 (since (28-7)/3+1=8) and therefore there are 64x32=2048 neurons in the first hidden layer.

The next part is the one that confuses me. Since the next convolution has kernel size (5,5) and stride (2,2) and uses 64 filters, does this means that we apply 64 convolutions to each 8x8 image obtained in the first hidden layer? That will give 64x32=2048 images of size 2x2 and there will be 2048x4=8192 neurons in the second hidden layer. But the weight matrix of the next layer is of shape (256,200). Shouldn't it be of shape (8192,200)? What is happening here?

Rajith Thennakoon Rajith Thennakoon · Accepted Answer · 2020-02-03T02:50:48

I think this explanation will helpful. General formula for calculate the output feature is this

N_out = ([N_input + 2*padding - kernal_size]/stride) + 1

For first conv layer

N_out = ([28 + 2(0) - 7])/3)+1
N_out = 8

you get 8x8 image with 32 filters

For second conv layer

N_out = ([8 + 2(0) - 5])/2)+1
N_out = 2.5

as you can see output size is 2.5,since you are not specified padding,tensorflow use default padding as "VALID".check this link for source.

Now lets see what will happen if you use valid padding over 8x8 image with 5X5 kernal with stride 2. this is the ascii art

1 2 3 4 5 6 7 8
|_______|          <--- 1 st window
    |_______|      <--- 2 nd window
        |________  <--- 3rd window will drop(since valid padding)

So you endup with 2x2 image with 64 channels after 2nd layer,which is 256.

for example,

from keras import layers

m = Sequential()
m.add(Conv2D(32,kernel_size=(7,7),strides=(3,3),use_bias=False,input_shape=(28,28,1)))
m.add(Conv2D(64,kernel_size=(5,5),strides=(2,2),use_bias=False))
m.add(Flatten())
m.add(Dense(200,use_bias=False))
m.add(Dense(150,use_bias=False))
m.add(Dense(10,use_bias=False,activation='softmax'))

for layer in m.layers: 
    print(layer.output_shape)

output

(None, 8, 8, 32)
(None, 2, 2, 64)
(None, 256)
(None, 200)
(None, 150)
(None, 10)

How keras manages weights of convolutional layers?

1 Answers