Fine-Tuning the Inception model in TensorFlow

Question

I want to use the pre-trained Inception model on my own data-set AND I also want to fine-tune the variables of the Inception model itself.

I have downloaded the pre-trained Inception model for TensorFlow from the following link:

http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz

I load the Inception model as follows:

graph = tf.Graph()
with graph.as_default():
    with tf.gfile.FastGFile('classify_image_graph_def.pb', 'rb') as file:
        graph_def = tf.GraphDef()
        graph_def.ParseFromString(file.read())
        tf.import_graph_def(graph_def, name='')

(Side note on the API: It would be nice if I could just write graph = tf.load_graph('inception.pb') instead of these six nested and complicated lines.)

Then I get a reference to the tensor for the last layer before the softmax-classifier in the Inception model:

last_layer = graph.get_tensor_by_name('pool_3:0')

Now I want to append a new softmax-classifier to the graph so I can train the new softmax-classifier AND train some or all of the variables in the Inception model. This is what I understand to be fine-tuning, as opposed to transfer learning where only the new softmax-classifier is trained on my own data-set.

I then use PrettyTensor to append the new softmax-classifier (note that y_true is a placeholder variable):

with pt.defaults_scope(activation_fn=tf.nn.relu):
    y_pred, loss = pt.wrap(last_layer).\
        flatten().\
        softmax_classifier(class_count=10, labels=y_true)

But this gives a long error-message where the last part reads:

ValueError: Tensor("flatten/reshape/Const:0", shape=(2,), dtype=int32) must be from the same graph as Tensor("pool_3:0", shape=(1, 1, 1, 2048), dtype=float32).

So I am apparently not allowed to combine two graphs like this.

I have also tried using a reshape() instead of flatten() as follows (note the last layer of the Inception model has 2048 features):

with pt.defaults_scope(activation_fn=tf.nn.relu):
    y_pred, loss = pt.wrap(last_layer).\
        reshape([-1, 2048]).\
        softmax_classifier(class_count=10, labels=y_true)

But this gives almost the same error:

ValueError: Tensor("reshape/Const:0", shape=(2,), dtype=int32) must be from the same graph as Tensor("pool_3:0", shape=(1, 1, 1, 2048), dtype=float32).

I have also tried wrapping it in a graph.as_default() like so:

with graph.as_default():
    with pt.defaults_scope(activation_fn=tf.nn.relu):
        y_pred, loss = pt.wrap(last_layer).\
            reshape([-1, 2048]).\
            softmax_classifier(class_count=10, labels=y_true)

But this gives a similar error:

ValueError: Tensor("ArgMax_1:0", shape=(?,), dtype=int64) must be from the same graph as Tensor("cross_entropy/ArgMax:0", shape=(1,), dtype=int64).

How would I do fine-tuning of the Inception model? I want to add a new softmax-classifier AND I want to fine-tune some or all of the variables in the Inception model itself.

Thanks!

EDIT:

I have a partial solution to the problem.

The error messages were because I didn't put all the code inside the with graph.as_default(): block. Putting all the code inside that block fixes the error-messages and I can now append a new softmax-layer to the Inception model using PrettyTensor, as described above.

However, the Inception model is apparently a 'frozen' graph which means that all variables have been converted to constants before it was saved.

So my question is now, whether I can somehow 'unfreeze' the graph for the Inception model, so I can continue training some or all of the variables of its graph? How would I do this?

Or should I instead use the new MetaGraph functionality?

https://www.tensorflow.org/versions/r0.11/how_tos/meta_graph/index.html

Where can I download a pre-trained MetaGraph for the Inception model?

chrisrn chrisrn · Accepted Answer · 2016-10-28T14:19:09

I have exactly the same question too. First of all you can fine-tune the whole network according to this: https://github.com/tensorflow/models/tree/master/inception#adjusting-memory-demands I figured out how to implement this code on my own dataset. The problem is that I want to fine-tune the whole network and then substitute the last 2 inception modules with a new conv layer and a softmax. However, I think that if I do the whole fine-tuning, I then need to restore weights until the 8th inception module and train only the new layers, but this can't happen only by a restore function from tensorflow. Another good trick is here: https://www.tensorflow.org/versions/r0.9/how_tos/image_retraining/index.html where you can take the pretrained network from .pb file, but this is good only for transfer learning according to tensorflow. The only solution came to my mind was the fine-tuning of the whole network and then export it to .pb file, in order to take the fine-tuned weights until the layer I want but yet I can't implement this. Generally tensorflow makes not clear how can we do this. I will post a question at git too. If anyone knows exactly what we can do, please answer.

Fine-Tuning the Inception model in TensorFlow

2 Answers