tf.Estimator.predict() issue when using a Tensorflow Hub module as the basis of a custom tf.Estimator

Question

I am trying to create a custom tensorflow tf.Estimator. In the model_fn passed to the tf.Estimator, I am importing the Inception_V3 module from Tensorflow Hub.

Problem: After fine-tuning the model (using tf.Estimator.train), the results obtained using tf.Estimator.predict are not as good as expected based on tf.Estimator.evaluate (This is for a regression problem.)

I am new to Tensorflow and Tensorflow Hub, so I could be making lots of rookie mistakes.

When I run tf.Estimator.evaluate() on my validation data, the reported loss is in the same ball park as the loss after tf.Estimator.train() was used to train the model. The problem comes in when I try to use tf.Estimator.predict() on the same validation data.

tf.Estimator.predict() returns predictions which I then use to calculate the same loss metric (mean_squared_error) which is computed by tf.Estimator.evaluate(). I am using the same set of data to feed to the predict function as the evaluate function. But I do not get the same result for the mean_squared_error -- not remotely close! (The mse I calculate from predict is much worse.)

Here is what I have done (edited out some details)... Define a model_fn with Tensorflow Hub module. Then call the tf.Estimator functions to train, evaluate and predict.

def my_model_fun(features, labels, mode, params):
    # Load InceptionV3 Module from Tensorflow Hub
    iv3_module =hub.Module("https://tfhub.dev/google/imagenet/inception_v3/feature_vector/1",trainable=True, tags={'train'})     

    # Gather the variables for fine-tuning
    var_list = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,scope='CustomeLayer')
    var_list.extend(tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,scope='module/InceptionV3/Mixed_5b'))

    predictions = {"the_prediction" : final_output}
    if mode == tf.estimator.ModeKeys.PREDICT:
        return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)

    # Define loss, optimizer, and evaluation metrics
    loss = tf.losses.mean_squared_error(labels=labels, predictions=final_output)
    optimizer =tf.train.AdadeltaOptimizer(learning_rate=learn_rate).minimize(loss, 
    var_list=var_list, global_step=tf.train.get_global_step())   
    rms_error = tf.metrics.root_mean_squared_error(labels=labels,predictions=predictions["the_prediction"])
    eval_metric_ops = {"rms_error": rms_error}

    if mode == tf.estimator.ModeKeys.TRAIN:
        return tf.estimator.EstimatorSpec(mode=mode, loss=loss,train_op=optimizer)

    if mode == tf.estimator.ModeKeys.EVAL:
        tf.summary.scalar('rms_error', rms_error)
        return tf.estimator.EstimatorSpec(mode=mode, loss=loss,eval_metric_ops=eval_metric_ops)  


iv3_estimator = tf.estimator.Estimator(model_fn=iv3_model_fn)   
iv3_estimator.train(input_fn=train_input_fn, steps=TRAIN_STEPS) 
iv3_estimator.evaluate(input_fn=val_input_fn)  

ii =0 
for ans in iv3_estimator.predict(input_fn=test_input_fn):
    sqErr = np.square(label[ii] - ans['the_prediction'][0])
    totalSqErr += sqErr
    ii += 1                           
mse = totalSqErr/ii

I expect that the mse loss reported by tf.Estimator.evaluate() should be the same as the when I calculate mse from the known labels and the output of tf.Estimator.predict()

Do I need to import the Tensorflow Hub model differently when I use predict? (use trainable=False in the call to hub.Module()?

Are the weights obtained from training being used when tf.Estimator.evaluate() runs, but not when tf.Estimator.predict()- runs?

other?

arnoegw arnoegw · Accepted Answer · 2019-02-18T09:57:05

There's a few things that seem to be missing from the code snippet. How is final_output computed from iv3_module? Also, mean squared error is an unusual choice of loss function for a classification problem; the common approach is to pass image features from the module into a a linear output layer with scores for each class ("logits") and a "softmax cross-entropy loss". For an explanation of these terms, you can review online tutorials like https://developers.google.com/machine-learning/crash-course/ (all the way to multi-class neural nets).

Regarding TF-Hub technicalities:

The variables of a Hub module are automatically added to the GLOBAL_VARIABLES and TRAINABLE_VARIABLES collections (if trainable=True, as you already do). No manual extension of those collections should be needed.
hub.Module(..., tags=...) should be set to {"train"} for mode==TRAIN and set to None or the empty set otherwise.

In general, it's useful to get a solution working end-to-end for your problem without fine-tuning as a baseline, and then add fine-tuning.

tf.Estimator.predict() issue when using a Tensorflow Hub module as the basis of a custom tf.Estimator

1 Answers