Why is my model overfitting after doing regularization and batchnormalization

Question

This is my CNN model structure.

def make_dcnn_model():
  model = models.Sequential()
  model.add(layers.Conv2D(5, (5, 5), input_shape=(9, 128,1), padding='same', strides = (1,2), activity_regularizer=tf.keras.regularizers.l1(0.001)))
  model.add(layers.LeakyReLU())
  model.add(BatchNormalization())

  model.add(layers.AveragePooling2D((4, 4), strides = (2,4)))

  model.add(layers.Conv2D(10, (5, 5),  padding='same', activity_regularizer=tf.keras.regularizers.l1(0.001)))
  model.add(layers.LeakyReLU())
  model.add(BatchNormalization())

  model.add(layers.AveragePooling2D((2, 2), strides = (1,2)))

  model.add(layers.Flatten())
  model.add(layers.Dense(50, activity_regularizer=tf.keras.regularizers.l1(0.001)))
  model.add(layers.LeakyReLU())
  model.add(BatchNormalization())

  model.add(layers.Dense(6, activation='softmax'))
  return model

The result shows that this model fit well the training data and for the validation data the great fluctuation of validation accuracy occurred.

Train on 7352 samples, validate on 2947 samples Epoch 1/3000 7352/7352 [==============================] - 3s 397us/sample - loss: 0.1016 - accuracy: 0.9698 - val_loss: 4.0896 - val_accuracy: 0.5816 Epoch 2/3000 7352/7352 [==============================] - 2s 214us/sample - loss: 0.0965 - accuracy: 0.9727 - val_loss: 1.2296 - val_accuracy: 0.7384 Epoch 3/3000 7352/7352 [==============================] - 1s 198us/sample - loss: 0.0930 - accuracy: 0.9727 - val_loss: 0.9901 - val_accuracy: 0.7855 Epoch 4/3000 7352/7352 [==============================] - 2s 211us/sample - loss: 0.1013 - accuracy: 0.9701 - val_loss: 0.5319 - val_accuracy: 0.9114 Epoch 5/3000 7352/7352 [==============================] - 1s 201us/sample - loss: 0.0958 - accuracy: 0.9721 - val_loss: 0.6938 - val_accuracy: 0.8388 Epoch 6/3000 7352/7352 [==============================] - 2s 205us/sample - loss: 0.0925 - accuracy: 0.9743 - val_loss: 1.4033 - val_accuracy: 0.7472 Epoch 7/3000 7352/7352 [==============================] - 1s 203us/sample - loss: 0.0948 - accuracy: 0.9740 - val_loss: 0.8375 - val_accuracy: 0.7998

Use regularisation and batch normalisation do not prevent over-fitting as such, but they will help - if you run the module without them it will probably be even worse. Another good thing to do is early stopping - periodically save the model and take the best according to validation accuracy. Over-fitting is a complex issue, there is no single way to solve it. — BenedictWilkinsAI

ramez ramez · Accepted Answer · 2019-10-29T12:31:56

Reducing overfitting is a matter of trial and error. There are many ways to deal with it.

Try to add more data to the model or maybe augmenting your data if you're dealing with images. (very helpful)

Try reducing the complexity of the model by tweaking the parameters of the layers.

Try stopping the training earlier.

Regularization and batch normalization are very helpful but it may be the case that your model is already performing much worse without them in terms of overfitting. Try different types of regularization. (maybe Dropout)

My guess is that by adding more variety in the data you're model will overfit less.

Why is my model overfitting after doing regularization and batchnormalization

1 Answers