Nan output when converting pretrained Tensorflow model to Tf-lite

Question

We have a problem when converting a pretrained Tensorflow model which we took as following files

snapshot_140.ckpt.index

snapshot_140.ckpt.meta

snapshot_140.ckpt.data-00000-of-00001

to tensorflow lite when we used the converted tflie file to make predictions , all predictions where Nan while the normal tensorflow model when loaded gave correct regression prediction

we got these files above, conversion from ckpt files to tensorflow .pb graph takes place via tensorflow.python.tools.freeze_graph tool

then TFlite_converter tool for converting to tflite

.pb works well when made into a tensorflow model.. but tflite model produces all nan output

one (potential solution) is inspired from this issue: https://github.com/tensorflow/tensorflow/issues/22803?fbclid=IwAR0x25CEZjHAQ0dUFBVWa1V3lVcCVIpT4q7lbUUGS9T59w2aXqnUVtNCBHI#issuecomment-474249701

so how to set batch_norm is_training flag to false given only ckpt files with no access to model architecture ?

if this is really the solution to the problem

or what should we do to fix the tensorflow lite model to give the same outputs as the tensorflow model

this code was used to get the tensorflow lite model which gave nans

    import tensorflow as tf
    with tf.Session(config=tf.ConfigProto(allow_soft_placement=True, 
    log_device_placement=True)) as sess:

        saver = tf.train.import_meta_graph('snapshot_140.ckpt.meta')

        saver.restore(sess, "snapshot_140.ckpt")
        inp = tf.get_default_graph().get_tensor_by_name("tower_0/Placeholder:0")
        out = tf.get_default_graph().get_tensor_by_name("tower_0/out/BiasAdd:0")
        print(out)
        converter = tf.contrib.lite.TFLiteConverter.from_session(sess, [inp], [out])
        tflite_model = converter.convert()
        open("converted_model.tflite", "wb").write(tflite_model)

Does the model have BatchNorm layers? The fix might not apply if it does not. One thing you could do is visualize your tflite graph to see if all layers/nodes seem okay (tensorflow.org/lite/guide/faq). Otherwise, there might be a bug in your inference code that runs the tflite graph. Could you post a snippet? — Sachin Joglekar
yes it has several BatchNorm Layers we visualized the graph and found nothing fishy till now , this is inference code for tflite: gist.github.com/ZyadShokry/a6157e6e6670753f4c98447661753251 — Kareem Abdel Salam
Did you solve this? I am having a similar issue. The model I use use a tf-slim resnet. I wonder if this is coming from tf-slim to tflite... — Nick Skywalker

Vikram Murthy Vikram Murthy · Accepted Answer · 2021-01-12T07:29:23

basically without access to the code /architecture i am not sure how one can even run the inference ! and if you cant run inference then what good are the checkpoints :) ..here's how i solved it ..i did have access to the full code (EAST text detection) and

restored the pre trained checkpoints and ran one epoch during which i ensured that i set the training flag to FALSE (note that one epoch is too short to meddle with variables that have been set after running enough epochs to create a baseline checkpoint ..so its harmless)
once the epoch completed, i simply saved the model using simple_save and it generated a pb file
used the regular tflite converter and in my inference file followed simple code from the tensorflow page to set input tensors, invoke and extract values from output tensors
the inference was as good as what i would have gotten via the other modes of restoring model and inferring ..basically lossless
the only way i could have made this process simpler would have been if i could convert the ckpt files directly to pb (using bazel and freeze graph) BUT that would have gotten me to the same place while inferring since the training flag would have remained unchanged

apologize if this workaround doesnt help

Nan output when converting pretrained Tensorflow model to Tf-lite

1 Answers