Test Accuracy Increases Whilst Loss Increases

Question

I have a deep network of only fully connected/dense layers with the shape 128-256-512-1024-1024 all layers use LeakyReLU activation, with no dropout and the final layer has a softmax activation.

During training after the 20th epoch the validation/test loss starts to reverse and go up but the test accuracy continues to increase also. How does this make sense? And is the test accuracy actually accurate if it were shown new data or is there some kind of false positive going on here?

I compiled the model like so:

model.compile(
    loss='categorical_crossentropy',
    optimizer='adam',
    metrics=['categorical_accuracy']
 )

Graphs of my train/test accuracy and loss curves:

Accuracy

Loss

Edit:

This may help. It's the true labels plotted against the predicted labels for the last epoch:

Thanks for the extra info. Given this, your comment on my answer is correct -- with a simple binary classification, my answer doesn't apply. Unfortunately, I don't think I can explain the effect without taking time to step through the Minimal, complete, verifiable example that is not in the question. — Prune
BTW, a scatter plot for the data would be a lot easier to read. Physical placement in the training set doesn't seem to matter, and the various crossing lines are hard to read. Also, how did you get values other than 0 or 1? — Prune

Prune Prune · Accepted Answer · 2018-03-07T19:39:16

This is easily possible with a loss function that is sensitive to the distance between an incorrect prediction and the ground truth. You can get 90% of the predictions correct, but if the misses are ridiculously far off the mark, your loss value can increase. This results in some models from not accurately identifying one or two critical factors in the ground truth.

Test Accuracy Increases Whilst Loss Increases

1 Answers