keras custom metric function how to feed 2 model outputs to a single metric evaluation function

Question

I have an CNN object detection model which has two heads(outputs) with tensor names 'classification' and 'regression'.

I want to define a metric function that accepts both the outputs at the same time, so that it looks into the regression predictions to decide which indexes to retain and use those indexes to select tensors from classification predictions and calculate some metric.

my current metric function defined with help from this link:

from tensorflow.python.keras.metrics import MeanMetricWrapper

class Accuracy2(MeanMetricWrapper):

    def __init__(self, name='dummyAccuracy', dtype=None):
        super(Accuracy2, self).__init__(metric_calculator_func, name, dtype=dtype)
        self.true_positives = self.add_weight(name='lol', initializer='zeros')

    @classmethod
    def from_config(cls, config):
        if 'fn' in config:
          config.pop('fn')
        return super(Accuracy2, cls).from_config(config)


    def update_state(self, y_true, y_pred, sample_weight=None):
      print("==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===")
      print("Y-True {}".format(y_true))
      print("Y-Pred {}".format(y_pred))
      print("==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===")

      update_ops = [self.true_positives.assign_add(1.0)]
      return tf.group(update_ops)

    def result(self):
      return self.true_positives

    def reset_states(self):
      # The state of the metric will be reset at the start of each epoch.
      self.true_positives.assign(0.)

which I call the during model compilation as :

training_model.compile(
    loss={
        'regression'    : regression_loss(),
        'classification': classification_loss()
    },
    optimizer=keras.optimizers.Adam(lr=lr, clipnorm=0.001),
    metrics=[Accuracy2()]
)

screen log during tf.estimator.train_and_evaluate is:

INFO:tensorflow:loss = 0.0075738616, step = 31 (11.941 sec)

INFO:tensorflow:global_step/sec: 4.51218

INFO:tensorflow:loss = 0.01015341, step = 36 (1.108 sec)

INFO:tensorflow:Saving checkpoints for 40 into /tmp/tmpcla2n3gy/model.ckpt.

INFO:tensorflow:Calling model_fn. ==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@=== Tensor("IteratorGetNext:1", shape=(?, 120087, 5), dtype=float32, device=/device:CPU:0) Tensor("regression/concat:0", shape=(?, ?, 4), dtype=float32) ==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@=== ==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@=== Tensor("IteratorGetNext:2", shape=(?, 120087, 2), dtype=float32, device=/device:CPU:0) Tensor("classification/concat:0", shape=(?, ?, 1), dtype=float32) ==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2019-06-24T08:20:35Z INFO:tensorflow:Graph was finalized. 2019-06-24 13:50:36.457345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 2019-06-24 13:50:36.457398: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-06-24 13:50:36.457419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 2019-06-24 13:50:36.457425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N 2019-06-24 13:50:36.457539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9855 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)

INFO:tensorflow:Restoring parameters from /tmp/tmpcla2n3gy/model.ckpt-40

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Evaluation [10/100]

INFO:tensorflow:Evaluation [20/100]

INFO:tensorflow:Evaluation [30/100]

INFO:tensorflow:Evaluation [40/100]

INFO:tensorflow:Evaluation [50/100]

INFO:tensorflow:Evaluation [60/100]

INFO:tensorflow:Evaluation [70/100]

INFO:tensorflow:Evaluation [80/100]

INFO:tensorflow:Evaluation [90/100]

INFO:tensorflow:Evaluation [100/100]

INFO:tensorflow:Finished evaluation at 2019-06-24-08:20:44

INFO:tensorflow:Saving dict for global step 40: _focal = 0.0016880237, _smooth_l1 = 0.0, dummyAccuracy = 100.0, global_step = 40, loss = 0.0016880237

This line :

==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===
Tensor("IteratorGetNext:1", shape=(?, 120087, 5), dtype=float32, device=/device:CPU:0)
Tensor("regression/concat:0", shape=(?, ?, 4), dtype=float32)
==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===
==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===
Tensor("IteratorGetNext:2", shape=(?, 120087, 2), dtype=float32, device=/device:CPU:0)
Tensor("classification/concat:0", shape=(?, ?, 1), dtype=float32)
==@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@===

shows that Accuracy2() is invoked two times first for regression then for classification. but I want that it gets invoked once with regression and classification fed into it together

Answer 1

If you need both y_true values for the metric.

In this case we will need to flatten our data so we can concat it in a single array. This will need fixed size outputs.

Assuming you have regOut and classOut as tensors. If they are 2D, just concatenate them, else:

regOut = Flatten()(regOut) #only if regOut is 3D or more
classOut = Flatten()(classOut) #only if classOut is 3D or more

out = Concatenate()([regOut,classOut])

Make the model with this single output:

model = Model(inputs, out)

Do the same with your datasets:

y_reg_train = y_reg_train.reshape((y_reg_train.shape[0], -1))
y_class_train = y_clas_trains.reshape((y_class_train.shape[0], -1))
y_train = np.concatenate([y_reg_train, y_class_train], axis=-1)

#same for y_val

Then create a metric that separates the two:

def metric(y_true, y_pred):

    reg_true = y_true[:,:flattened_size_of_reg]
    class_true = y_true[:, flattened_size_of_reg:]
    
    reg_pred = y_pred[:,:flattened_size_of_reg]
    class_pred = y_pred[:, flattened_size_of_reg:]

    #calculate the metric

    return value

Train with the combined output:

model.fit(x_train, y_train, ...)

Answer 2

If you don't need y_true values in that metric

This is an ugly answer, but....

You have to make a layer to calculate the metric for you. Use a Lambda:

Given that regOut and classOut are your output tensors, in model creation, instead of creating a model like Model(inputs, [regOut,classOut]) you will:

def metricFunc(modelOutputs):
    regressionOutput = modelOutputs[0]
    classOutput = modelOuptuts[1]

    #calculate metric
    return calculatedMetric


metricTensor = Lambda(metricFunc, name='metric_layer')([regOut,classOut])

Make the metric be an output to the model:

model = Model(inputs, [regOut, classOut, metricTensor])

Create a dummy loss and a dummy metric for compilation:

def dummyLoss(true,pred):
    return K.zeros(K.shape(true)[:1])

def dummyMetric(true,pred):
    return pred

In compilation:

model.compile(loss = [regLoss, classLoss, dummyLoss], 
              metrics={'metric_layer':dummyMetric}, 
              optimizer=...)

This requires that you train with a dummy tensor for metricTensor too:

model.fit(x_train, [y_reg,y_class,np.zeros(y_reg.shape[:1])], ...)

Answer 3

Let me show you an elegant way to achieve this.

• First of all, define an outer function that wraps your metric, so that you can pass your regression tensor reg_out:

  def metric_func(reg_out):
      def metric(y_true, class_out):
          return your_metric(reg_out, class_out, y_true)
      return metric
  

• Next, name your classification tensor class_out by setting the argument name of the layer that produces it. For example:

  class_out = Dense(1, name='class_out')(something)
  

• Finally, set the argument metrics of model.compile as follows:

  model.compile(...,
                metrics={'class_out': metric_func(reg_out)})