Keras callback ModelCheckpoint doesn't save weights

3
votes

Solution suggested here did not solve my problem this question.

I'm trying to use callbacks ModelCheckpoint and EarlyStopping to save the best weights when early stopping. After the first epoch I get a runtime warning but the codes runs the remaining epochs without errors, but still no file containing weight appears. The warning after the first epoch is the following:

RuntimeWarning: Can save best model only with val_acc available, skipping. 'skipping.' % (self.monitor), RuntimeWarning) RuntimeWarning: Early stopping conditioned on metric val_acc which is not available. Available metrics are: val_loss,val_accuracy,loss,accuracy (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning

I have validation data added to the fit() function so I'm not sure why.

filepath = "weights_best.hdf5"

model.compile(loss="mean_squared_error",
                  metrics=['accuracy'],
                  optimizer=optimizer)
batchSize = 64 
numEpochs = 75
validation_data = (data.x_valid, data.y_valid)

callbackCheckpoint = keras.callbacks.callbacks.ModelCheckpoint(filepath,
                                                                   monitor='val_acc',
                                                                   save_best_only=True,
                                                                   save_weights_only= True,
                                                                   mode='max')

callbackEarlyStop = keras.callbacks.callbacks.EarlyStopping(monitor='val_acc',
                                                                min_delta=0,
                                                                patience=7,
                                                                verbose=0,
                                                                mode='auto')
callbacks = [callbackCheckpoint, callbackEarlyStop]
model.fit(data.x_train, data.y_train, batchSize, numEpochs, callbacks=callbacks,
              validation_data=validation_data)

Any help would be greatly appreciated!

2
pythonkerasdeep-learningconv-neural-network
What's the output when you run fit? If val_acc is there then I'd suggest you change the Keras and Tensorflow version, the code doesn't look like there's any problem. – Natthaphon Hongcharoen
At the end of each epoch it still outputs 45000/45000 [==============================] - 126s 3ms/step - loss: 0.1793 - accuracy: 0.0993 - val_loss: 0.1806 - val_accuracy: 0.0968, so yes, val_acc is present! I'll give it a try. – JustAnotherUser
Nooooooo, that's a val_accuracy not val_acc. Change that!! – Natthaphon Hongcharoen
Oh, they are different? What should I change? – JustAnotherUser
monitor='val_acc' to monitor='val_accuracy' or metrics=['accuracy'] to metrics=['acc'] – Natthaphon Hongcharoen

2 Answers

6
votes

Change monitor='val_acc' to monitor='val_accuracy' or metrics=['accuracy'] to metrics=['acc']

0
votes

Yes monitor 'val_accuracy', Just a comment, monitoring and saving the weights based on the highest val_accuracy may not give you the "best" answer. Monitoring 'val_loss' is usually a better answer. val_loss and val_accuracy are calculated in entirely different ways. It is not uncommon to see the val_loss increase but have val_accuracy increase as well or vice versa. In the end you want to use a model with the lowest validation loss. If you do do that make sure to change mode='min' or mode='auto in the checkpoint callback.