Testing a saved Convolutional autoencoder

Question

I have trained and saved a Convolutional Autoencoder in Keras. Before saving the model in .h5 format, it had a training loss of 0.2394 and a validation loss of 0.2586. In testing the saved model, I get a loss which is more than double the validation loss, 0.6707. I am actually testing it with the sample from the training data just to see if I will get the same loss or even closer as it was during training.

Here is how I calculate the loss, where 'total' is the total number of images I am passing to test the model

score = np.sqrt(metrics.mean_squared_error(predicteds,images))
print ('Loss:',score/total)

Am I making a mistake on how I calculate the test loss?

Here is model compile

autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

and train verbose

Epoch 18/20 167/167 [==============================] - 462s 3s/step - loss: 0.2392 - val_loss: 0.2585

Epoch 19/20 167/167 [==============================] - 461s 3s/step - loss: 0.2399 - val_loss: 0.2609

Epoch 20/20 167/167 [==============================] - 475s 3s/step - loss: 0.2394 - val_loss: 0.2586

Answer 1

I think you are confusing the metrics and the loss function.

Based on your model.compile(), you are using the binary_crossentropy loss function. This means that the loss mentioned in the verbose is related to the binary crossentropy (both loss - training loss and val_loss - validation loss).

You are scoring the model using RMSE, but then comparing the RMSE to binary crossentropy loss.

To train using MSE or to make use of other comparable metrics, you need to compile the model with the MSE loss or make use of MSE as a metric. For more information on keras.losses and keras.metrics, have a look at the documentation.

Answer 2

Assuming that you complied with autoencoder.compile(optimizer='adam', loss='mean_squared_error'). But it seens that you use root mean square error in np.sqrt(metrics.mean_squared_error(predicteds,images)) to calculate the loss rather than mean square error. When a number is less than 1, its square root is larger than itself. Maybe this is why your test loss is strangely larger than you training loss. By the way, you can use autoencoder.evaluate(images, images) to get the test loss.