Fluctuating Learning Curve

Question

I am training a CNN model, here is the code.

model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(300, 300,3)),
    tf.keras.layers.MaxPooling2D(2, 2),
    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2, 2),
    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2, 2),
    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    tf.keras.layers.Flatten(input_shape=(300, 300)),
    tf.keras.layers.Dense(512, activation=tf.nn.relu),
    tf.keras.layers.Dense(256, activation=tf.nn.relu),
    tf.keras.layers.Dense(64, activation=tf.nn.relu),
    tf.keras.layers.Dense(1, activation='linear')
])

I am using Adam with 0.00003 learning rate, and training on 100 epochs.

However, The 10 epochs the validation starts to fluctuate between 0.16 and 0.22. ( I can't use early stopping because each time i retry the training the minimum is reached after a random number of epochs ).

Is this learning Curve normal ? what can i do to improve it ?

from my experience, this is absolutely normal, no need to worry. You can try pushing your training a little longer (around 500 epochs) and lowering your learning rate (divide it by 3 and then 10) to see if you get better results. — Joseph Budin
You can also try to add BatchNorms. i don't know what problem you are trying tosolve, but with Conv2D, it's usually a good idea — Joseph Budin
I tried with 300 epochs the validation did'nt change i can see that i am overfitting, as only the test decrease to 0. I tried both Dropout and BatchNormalization but the error increased it become between 0.18-0.28 — Wassim Bouatay
OK, then I guess you an still try to lower your learning rate, but once again, it is perfectly normal to have this kind of oscillations (but they can be a symptom of a too high learning rate) — Joseph Budin

Imtinan Azhar Imtinan Azhar · Accepted Answer · 2019-07-24T10:35:38

Yes, it is normal, no need to get worried, but for the future, I would suggest applying smoothing to the graphs, this will help you understand a lot better what is going on. For example, look at this graph I captured from tensorboard

The transparent graph is with the initial values, and the darker graph is a smoothened graph. the smoothened graph shows that the model is minimizing the loss, very haphazardly and slowly, but it is minimizing nonetheless.

Sometimes viewing a smoothened out graph can help better identify the training of the models.

Fluctuating Learning Curve

1 Answers