Why does loss decrease but accuracy decreases too (Pytorch, LSTM)?

0
votes

I have built a model with LSTM - Linear modules in Pytorch for a classification problem (10 classes). I am training the model and for each epoch I output the loss and accuracy in the training set. The ouput is as follows:

epoch: 0 start!
Loss: 2.301875352859497
Acc: 0.11388888888888889
epoch: 1 start!
Loss: 2.2759320735931396
Acc: 0.29
epoch: 2 start!
Loss: 2.2510263919830322
Acc: 0.4872222222222222
epoch: 3 start!
Loss: 2.225804567337036
Acc: 0.6066666666666667
epoch: 4 start!
Loss: 2.199286699295044
Acc: 0.6511111111111111
epoch: 5 start!
Loss: 2.1704766750335693
Acc: 0.6855555555555556
epoch: 6 start!
Loss: 2.1381614208221436
Acc: 0.7038888888888889
epoch: 7 start!
Loss: 2.1007182598114014
Acc: 0.7194444444444444
epoch: 8 start!
Loss: 2.0557992458343506
Acc: 0.7283333333333334
epoch: 9 start!
Loss: 1.9998993873596191
Acc: 0.7427777777777778
epoch: 10 start!
Loss: 1.9277743101119995
Acc: 0.7527777777777778
epoch: 11 start!
Loss: 1.8325848579406738
Acc: 0.7483333333333333
epoch: 12 start!
Loss: 1.712520718574524
Acc: 0.7077777777777777
epoch: 13 start!
Loss: 1.6056485176086426
Acc: 0.6305555555555555
epoch: 14 start!
Loss: 1.5910680294036865
Acc: 0.4938888888888889
epoch: 15 start!
Loss: 1.6259561777114868
Acc: 0.41555555555555557
epoch: 16 start!
Loss: 1.892195224761963
Acc: 0.3655555555555556
epoch: 17 start!
Loss: 1.4949012994766235
Acc: 0.47944444444444445
epoch: 18 start!
Loss: 1.4332982301712036
Acc: 0.48833333333333334

For loss function I have used nn.CrossEntropyLoss and Adam Optimizer.
Although the loss is constantly decreasing, the accuracy increases until epoch 10 and then begins for some reason to decrease.

Why is this happening ?

Even if my model is overfitting, doesn't that mean that the accuracy should be high ?? (always speaking for accuracy and loss measured on the training set, not the validation set)

1
neural-networkpytorchlstmloss-functioncross-entropy

1 Answers

1
votes

Decreasing loss does not mean improving accuracy always. I will try to address this for the cross-entropy loss.

CE-loss= sum (-log p(y=i))

Note that loss will decrease if the probability of correct class increases and loss increases if the probability of correct class decreases. Now, when you compute average loss, you are averaging over all the samples, some of the probabilities may increase and some of them can decrease, making overall loss smaller but also accuracy drops.