Custom loss function in Keras: Initializing and appending values to a Tensor in loop

Question

I am writing a custom loss function in Keras using Keras and Tensorflow backend functions. I want to minimize the mean square error between f(y_true) and f(y_pred), where f(y) is a nonlinear function.

f(y) = [f1(y) f2(y) ... f12(y)], and fk(y) = Xi_k*Theta_k(y), k = 1,2,...,12. Xi_k and Theta_k(y) are tensors of rank 1.

Since y_true and y_pred are of size (batchSize x 15), I need to calculate the value of f(y) over a loop for all the samples in the training batch (I believe avoiding the loop is not possible). The output of the loop operation would be tensors of size (batchSize x 12)

[[f(y_true[1,:])],[f(y_true[2,:])],...,[f(y_true[batchSize,:])]]

and

[[f(y_pred[1,:])],[f(y_pred[2,:])],...,[f(y_pred[batchSize,:])]]

Generally, when dealing with arrays or matrix, we initialize a matrix of the desired size and assign values to it in the loop or we create an empty matrix and append values to it in the loop. But how do we do the same with tensors?

Below is a simplified form of the custom loss function (just calculating f1(y_true) and f1(y_pred)). The initialization and appending function don't work as they are not tf/Keras operations, what should I use in place of these to make it work with tensor?

matd = spio.loadmat('LPVmodel_iVarDeg.mat', squeeze_me=True) 
mate = spio.loadmat('PNLSS_model_modified.mat', squeeze_me=True)

def custom_loss_fn(y_true, y_pred):
    iVarDeg1 = tf.convert_to_tensor(matd['iVarDeg1'], dtype=tf.float32) # (XiSize(1) x 15)
    Xi1 = tf.convert_to_tensor(mate['Xim1'], dtype=tf.float32) # (XiSize(1) x 1) 

    batchSize = m 
    fy_true = [] # initialization
    fy_pred = [] # initialization

    for i in range(batchSize):
        yin = y_true[i,:]  # (1 x 15) network output
        tin = y_pred[i,:]  # (1 x 15) target
    
        ypowerD = tf.math.pow(yin,iVarDeg1) # element wise power (XiSize(1)x15)
        monomial = tf.math.reduce_prod(ypowerD) # column wise product of elements (XiSize(1)x1)
        Theta1 = monomial  # nonlinear basis for state eq 1 (XiSize(1)x1)
    
        ypowerD = tf.math.pow(tin,iVarDeg1)
        monomial = tf.math.reduce_prod(ypowerD)
        Gamma1 = monomial

        temp = tf.math.reduce_sum(tf.math.multiply(Xi1,Theta1)) # sum(element wise prod) 
        fy_true.append(temp)
    
        temp = tf.math.reduce_sum(tf.math.multiply(Xi1,Gamma1))
        fy_pred.append(temp)
    

    return Kb.mean(Kb.sum(Kb.square(fy_pred - fy_true))

you say that python list appending function doesn't work as they are not tf/Keras operations". if you are using Eager execution (tf.2.x) and your method isn't decorated by @function the python list will work fine in a loop. — Tou You
@TouYou I am not aware of eager execution, can you please share some documents/examples. I am new to Tensorflow and Keras. I first wrote the custom loss function using NumPy operations, but it didn't work. Model.compile() would run for hours and eventually stop responding. I was later told that I need to use tf/Keras operations to write the custom function. I replaced all the NumPy operations with tf/Keras operation but I couldn't figure out how to append the values of f(y) for each sample (calculated in the loop) to a tensor. — Sourav Sinha
Why do you want to use a loop, you can rewrite your operations using tensor_way for example: instead of : yin = y_true [i,:] ypowerD = tf.math.pow (yin, iVarDeg1) you can operate on all the tensor y_true: (change shape if necessary to allow broadcasting) ypowerD = tf.math.pow (y_true, iVarDeg1) — Tou You
@TouYou I am using a loop because iVarDeg1 is not a vector tensor. It's of size (np x 15), np > 1. tf.math.pow (yin, iVarDeg1) returns me a tensor of size (np x 15) whose elements are basically tf.math.pow (yin, iVarDeg1[k,:]). — Sourav Sinha

Tou You Tou You · Accepted Answer · 2020-08-09T13:13:17

in graph mode ,If you want to fill a tensor in a loop like a list,you can use TensorArray :

to 'initialise' e.g:

ta = tf.TensorArray(tf.float32, size=0, dynamic_size=True, clear_after_read=False)

to 'append' :

ta = ta.write(1, 20)

to transform TensorArray to tensor :

ta.stack()

Custom loss function in Keras: Initializing and appending values to a Tensor in loop

1 Answers