Keras validation loss and metric inconsistent

Question

RNN model used for regression, cf. Chollet, Deep Learning with Python, 6.3.1 A temperature-forecasting problem
In this example I used random data, both regressors and regressand
I have used the mean absolute error, both as loss function and as a metric
I do not understand the values I get for val_loss and val_mean_absolute_error. Neither of them make sense to me.

code:

import tensorflow as tf
import numpy as np
from keras.models import Sequential
from keras import layers
from keras.optimizers import Adam
import keras

I use random input data:

data_np = np.random.rand(6400,10)
target_np = np.random.rand(6400,)

Normalizing the data:

mean1 = data_np[:].mean(axis=0)
std1 = data_np[:].std(axis=0)  
    
data_np -= mean1
data_np /= std1
        
mean2 = target_np.mean(axis=0)
std2 = target_np.std(axis=0)
        
target_np -= mean2
target_np /= std2

Create RNN input with lookback:

lookback = 7

train_data = np.array([data_np[(i-lookback):i,:] for i in range(lookback,len(data_np))])
target_data = target_np[lookback:len(data_np)]

And then set up a simple RNN:

model = Sequential()
model.add(layers.SimpleRNN(64,
                     activation = 'relu',
                     return_sequences=False,
                     input_shape=(train_data.shape[1], train_data.shape[2])))
model.add(layers.Dense(1))

opt = Adam(learning_rate=0.1)

mae = tensorflow.keras.losses.MeanAbsoluteError()

model.compile(optimizer=opt, loss=mae, metrics=[mae])
history = model.fit(train_data, target_data,
                              steps_per_epoch=round(0.7*len(train_data))//64,
                              epochs=10,
                              shuffle=False,
                              validation_split=0.3,
                              validation_steps = round(0.3*len(train_data))//64,
                              verbose=1)

The output then looks like this:

Train on 3495 samples, validate on 1498 samples Epoch 1/10 54/54 [==============================] - 2s 38ms/step - loss: 0.7955 - mean_absolute_error: 0.7955 - val_loss: 0.0428 - val_mean_absolute_error: 22.6301 Epoch 2/10 54/54 [==============================] - 2s 30ms/step - loss: 0.7152 - mean_absolute_error: 0.7152 - val_loss: 0.0421 - val_mean_absolute_error: 22.2968

I would expect val_loss and val_mean_absolute_error to be the same. Moreover, the levels don't make much sense either. After 10 epochs, I get

Epoch 10/10 54/54 [==============================] - 2s 32ms/step - loss: 0.7747 - mean_absolute_error: 0.7747 - val_loss: 0.0409 - val_mean_absolute_error: 21.6337

If I calculate the mean absolute error manually:

N=len(data_np)

val_data = np.array([data_np[(i-lookback):i,:] for i in range(round(0.7*N),N)])
val_target = target_np[round(0.7*N):N]

model_output = model.predict(val_data)

model_output=[output[0] for output in model_output]
np.mean(abs(model_output-val_target))

0.940300949276649

This looks like a result that one could expect. However, it is not even close to either val_loss or val_mean_absolute_error. What is wrong here?

NOo07 NOo07 · Accepted Answer · 2021-03-25T21:24:30

OK. I managed to solve the issue by consistently using tensorflow.keras. So, replacing

import tensorflow as tf
import numpy as np
from keras.models import Sequential
from keras import layers
from keras.optimizers import Adam
import keras

with

import tensorflow as tf
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
import tensorflow.keras

(and corrected a couple of details in the original question)

Keras validation loss and metric inconsistent

1 Answers