tensorflow: model is not restoring the weights

Question

I'm experimenting with Text Summarization. I came across a github code which uses amazon food reviews dataset and Tensorflow 1.1.0 to write the code.

I ran the code by putting a while loop on inference code so that I could check multiple summaries and worked well.

checkpoint = "./best_model.ckpt"

loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # Load saved model
    loader = tf.train.import_meta_graph(checkpoint + '.meta')
    loader.restore(sess, checkpoint)

    input_data = loaded_graph.get_tensor_by_name('input:0')
    logits = loaded_graph.get_tensor_by_name('predictions:0')
    text_length = loaded_graph.get_tensor_by_name('text_length:0')
    summary_length = loaded_graph.get_tensor_by_name('summary_length:0')
    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')

    while True:
        input_sentence = input()
        text = text_to_seq(input_sentence)

        #Multiply by batch_size to match the model's input parameters
        answer_logits = sess.run(logits, {input_data: [text]*batch_size, 
                                      summary_length: [np.random.randint(5,8)], 
                                      text_length: [len(text)]*batch_size,
                                      keep_prob: 1.0})[0] 

        # Remove the padding from the tweet
        pad = vocab_to_int["<PAD>"] 

        print('Original Text:', input_sentence)

        print('\nText')
        print('  Word Ids:    {}'.format([i for i in text]))
        print('  Input Words: {}'.format(" ".join([int_to_vocab[i] for i in text])))

        print('\nSummary')
        print('  Word Ids:       {}'.format([i for i in answer_logits if i != pad]))
        print('  Response Words: {}'.format(" ".join([int_to_vocab[i] for i in answer_logits if i != pad])))

Some snippet of code to save the model:

checkpoint = "best_model.ckpt" 
with tf.Session(graph=train_graph) as sess:
    sess.run(tf.global_variables_initializer())
    # train the model
    if update_loss <= min(summary_update_loss):
        print('New Record!') 
        stop_early = 0
        saver = tf.train.Saver() 
        saver.save(sess, checkpoint)

The whole code can be found on the link given above.

When I stopped the model and rerun it but this time only commenting out the training the model part it didn't run at all well.

To confirm whether the weights are loading properly, I tried to restore the model and start training again where it left off. But the loss were terrible and almost equal to the loss when you start training freshly. That concluded me the weights are not properly loading to the model.

Then I tried to save the model with tf.saved_model.builder.SavedModelBuilder and tried to retrain where it left off, but same problem arises. It again gave loss as if you are training the model from the beginning.

Answer 1

You are loading only the meta file which doesn't hold values of variables. Use below code.

saver_path = 'path to your checkpoint'
checkpoint = tf.train.get_checkpoint_state(saver_path)
input_checkpoint = checkpoint.model_checkpoint_path
saver.restore(session, input_checkpoint)

Answer 2

Every time a tensorflow model state is saved following five files are created:

1. checkpoint
2. events.out.tfevents
3. model-xxxx.data.00000-0f-00001
4. model-xxxx.index, and
5. model-xxxx.meta

Let us assume that my model is stored in a directory named "saved_models" and all the above files are directly stored under it. You could restore your model to continue training where you left off, or you could also do a test run for evaluating your saved model on the test data.

import tensorflow as tf

checkpoint_dir = "saved_models"
with tf.compat.v1.Session() as sess:
    saver = tf.compat.v1.train.Saver()
    ckpt = tf.compat.v1.train_get_checkpoint_state(checkpoint_dir)
    saver.restore(sess, ckpt.model_checkpoint_path)
    saver.recover_last_checkpoints(ckpt.all_model_checkpoint_paths)
    print("Model checkpoint has been successfully restored.")
    # resume training or evaluate