Why does Keras' train_on_batch produce zero loss and accuracy at the second epoch?

Question

I am using a big dataset, and so I'm trying to use train_on_batch(or fit with epoch = 1)

model = Sequential()
model.add(LSTM(size,input_shape=input_shape,return_sequences=False))
model.add(Dense(output_dim))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=["accuracy"])

for e in range(nb_epoch):
    for batch_X, batch_y in batches:
        model.train_on_batch(batch_X,batch_y)
        # or
        # model.fit(batch_X,batch_y,batch_size=batch_size,nb_epoch=1,verbose=1,shuffle=True,)

But when training starts, this happens:

(0, 128)
Epoch 1/1
128/128 [==============================] - 2s - loss: 0.3262 - acc: 0.1130

(129, 257)
Epoch 1/1
128/128 [==============================] - 2s - loss: -0.0000e+00 - acc: 0.0000e+00

It doesn't matter how many epochs I wait, it doesn't change. Even If I change the batch size, the same thing happens: The first batch has good values and then it just goes to "loss: -0.0000e+00 - acc: 0.0000e+00" again.

Can someone maybe help in understanding what's happening here?

This might happen if your training data contains very small amount of unique examples and your network learn all of them in its first batches. Maybe you've accidentally put identical elements by using an array reference instead of copies in your dataset creation script. — Dmitry Kostyaev
Yeah, take a look at the predictions and labels and see if the network is actually getting 0 accuracy. That will help you debug. — 1''
@DmitryKostyaev Identical elements. It was a tiny mistake, I feel silly. Thank you for the help. — qkeras
Voted to close: (1) Keras has changed immensely since 4 years ago; (2) not enough debug details; (3) this is the only question on OP's account, so unlikely (2) is ever getting addressed. — OverLordGoldDragon

Burak Tozluk Burak Tozluk · Accepted Answer · 2020-05-13T16:32:01

if there are too many categories and if your data set is not good; When the system does not find a good result, it automatically prints the local minima. Try to change the learning rate.

opt = keras.optimizers.Adam(learning_rate=0.01)
model.compile(loss='categorical_crossentropy', optimizer=opt)

Why does Keras' train_on_batch produce zero loss and accuracy at the second epoch?

2 Answers