keras custom metrics for multi-label classification without all()

Question

I'm using sigmoid and binary_crossentropy for multi-label classification. A very similar question asked here. And the following custom metric was suggested:

from keras import backend as K

def full_multi_label_metric(y_true, y_pred):
    comp = K.equal(y_true, K.round(y_pred))
    return K.cast(K.all(comp, axis=-1), K.floatx())

But I do not want to use all() because for one single sample with a true label of [1, 0, 0, 1, 1] and a predicted label of [0, 0, 0, 1, 1] I do not consider the prediction accuracy as zero (due to the the fact that the labels for the last four classes have been predicted correctly).

Here is my model:

# expected input data shape: (batch_size, timesteps, data_dim)
model = Sequential()
model.add(Masking(mask_value=-9999, input_shape=(197, 203)))
model.add(LSTM(512, return_sequences=True))
model.add(Dense(20, activation='sigmoid')) 
model.compile(loss='binary_crossentropy',
              optimizer=SGD(lr=1e-3, decay=1e-4, momentum=0.9, nesterov=True),
              metrics = ['accuracy'])
print(model.summary())

Here is my y_pred for one example:

pred = model.predict(X_test)
y_pred = pred[0,196,:]
y_pred
array([2.6081860e-01, 9.9079555e-01, 1.4816311e-01, 8.6009043e-01,
       2.6759505e-04, 3.0792636e-01, 2.6738405e-02, 8.5339689e-01,
       5.1105350e-02, 1.5427300e-01, 6.7039116e-05, 1.7909735e-02,
       6.4140558e-04, 3.5133284e-01, 5.3054303e-02, 1.2765944e-01,
       2.9298663e-04, 6.3041472e-01, 5.8620870e-03, 5.9656668e-01],
      dtype=float32)

Here is my y_true for one example:

y_true = Y_test[0,0,:]
y_true
array([1., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 1., 0., 0.,
       0., 0., 1.])

My question is: How can I set a Keras custom metric function so that each element in y_pred should be compared to the each element in y_true, then an accuracy measure will be given during training? I want to use this metric in metrics = [X])?

Pedro Marques Pedro Marques · Accepted Answer · 2021-05-24T19:25:31

Unless I'm mistaken the default binary_crossentropy metric/loss already does what you need. Taking your example

import tensorflow as tf
from tensorflow import keras

y_true = tf.constant([[1, 0, 0, 1, 1]], dtype=tf.int32)
y_pred = tf.constant([[0.6, 0, 0, 1, 1]], dtype=tf.float32)

m = keras.metrics.binary_crossentropy(y_true, y_pred)
m.numpy()

The output is [-log(0.6) / 5].

i.e. The metric/loss for the batch takes into account the losses for each of the 20 outputs of the model. Which I'm assuming represent time steps.

As a metric, it is much more common to use binary_accuracy. Example:

y_true = tf.constant([[1, 0, 0, 1, 1]], dtype=tf.int32)
y_pred = tf.constant([[0.1, 0, 0, 1, 1]], dtype=tf.float32)
keras.metrics.binary_accuracy(tf.cast(y_true, tf.float32), y_pred)

One can get a better idea of the performance of the model via an ROC metric (https://www.tensorflow.org/api_docs/python/tf/keras/metrics/AUC) which measures the curve at various thresholds see an explanation at https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5. Personally, I tend to use an accuracy metric while training and look at the precision/recall curve after the model is trained in order to check that it behaves as expected and select the prediction threshold.

keras custom metrics for multi-label classification without all()

1 Answers