Keras Model - Functional API - adding layers to existing model

Question

I am trying to learn to use the Keras Model API for modifying a trained model for the purpose of fine-tuning it on the go:

A very basic model:

inputs = Input((x_train.shape[1:]))
x = BatchNormalization(axis=1)(inputs)
x = Flatten()(x)
outputs = Dense(10, activation='softmax')(x)
model1 = Model(inputs, outputs)
model1.compile(optimizer=Adam(lr=1e-5), loss='categorical_crossentropy', metrics=['categorical_accuracy'])

The architecture of it is

InputLayer -> BatchNormalization -> Flatten -> Dense

After I do some training batches on it I want to add some extra Dense layer between the Flatten one and the outputs:

x = Dense(32,activation='relu')(model1.layers[-2].output)
outputs = model1.layers[-1](x)

However, when I run it, i get this:

ValueError: Input 0 is incompatible with layer dense_1: expected axis -1 of input shape to have value 784 but got shape (None, 32)

Could someone please explain what is going on and how/if can I add layers to an already trained model?

Thank you

Daniel Möller Daniel Möller · Accepted Answer · 2018-02-02T15:03:08

A Dense layer is made strictly for a certain input dimension. That dimension cannot be changed after you define it (it would need a different number of weights).

So, if you really want to add layers before a dense layer that is already used, you need to make sure that the outputs of the last new layer is the same shape as the flatten's output. (It says you need 784, so your new last dense layer needs 784 units).

Another approach

Since you're adding intermediate layers, it's pointless to keep the last layer: it was trained specifically for a certain input, if you change the input, then you need to train it again.

Well... since you need to train it again anyway, why keep it? Just create a new one that will be suited to the shapes of your new previous layers.

Keras Model - Functional API - adding layers to existing model

1 Answers