I have a problem with TensorFlow where the performance of my LSTM drops dramatically (from 70% to <10%) when I use input dropout.
As I understand it I should set the input_keep_probability to (e.g.) 0.5 during training and then to 1 during testing. This makes perfect sense, but I cant get it to work as intended. If I set the dropout during testing the same as during training my performance improves (this is not the case in the example below, but this needs less code and the point is the same).
The accuracy and cost of 3 runs
The 'best' line is no dropout, the worst line is [keep_prob@train: 0.5, keep_prob@test: 1] and the center line is [keep_prob@train: 0.5, keep_prob@test: 0.5]. These are cost and accuracy on the test set. They all behave as expected on the train set.
Below is the code that I think is essential. Sadly I cant post the full code or a data sample due to its sensitive nature, but please comment if you need more info.
lstm_size = 512
numLayers = 4
numSteps = 15
lstm_cell = tf.nn.rnn_cell.LSTMCell(lstm_size, state_is_tuple=True, forget_bias=1)
lstm_cell = tf.nn.rnn_cell.DropoutWrapper(lstm_cell, input_keep_prob=input_keep_prob)
cell = tf.nn.rnn_cell.MultiRNNCell([lstm_cell] * numLayers, state_is_tuple=True)
_inputs = [tf.squeeze(s, [1]) for s in tf.split(1, numSteps, input_data)]
(outputs, state) = rnn.rnn(cell, _inputs, dtype=tf.float32)
outputs = tf.pack(outputs)
#transpose so I can put all timesteps through the softmax at once
outputsTranspose = tf.reshape(outputs, [-1, lstm_size])
softmax_w = tf.get_variable("softmax_w", [lstm_size, nof_classes])
softmax_b = tf.get_variable("softmax_b", [nof_classes])
logits = tf.matmul(outputsTranspose, softmax_w) + softmax_b
loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits, targets)
cost = tf.reduce_mean(loss)
targetPrediction = tf.argmax(logits, 1)
accuracy = tf.reduce_mean(tf.cast(tf.equal(targetPrediction, targets), "float"))
"""Optimizer"""
with tf.name_scope("Optimizer") as scope:
tvars = tf.trainable_variables()
#We clip the gradients to prevent explosion
grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars),maxGradNorm)
optimizer = tf.train.AdamOptimizer(learning_rate)
gradients = zip(grads, tvars)
train_op = optimizer.apply_gradients(gradients)
with tf.Session() as sess:
sess.run(init_op);
for i in range(nofBatches * nofEpochs):
example_batch, label_batch = sess.run(readTrainDataOp)
result = sess.run([train_op, accuracy, trainSummaries], feed_dict = {input_data: example_batch, targets: label_batch, input_keep_prob:trainInputKeepProbability, batch_size_ph:batch_size})
#logging
if i % 50 == 0:
runTestSet()
#relevant part of runTestSet():
#result = sess.run([cost, accuracy], feed_dict = {input_data: testData, targets: testLabels, input_keep_prob:testInputKeepProbability, batch_size_ph:testBatchSize})
#logging
What am I doing wrong that creates this unexpected behaviour?
EDIT: Here is an image of what an input sample looks like See next link.
The problem also presists with only 1 layer.
EDIT: I made an example that reproduces the problem. Just run the python script with the path to test_samples.npy as first argument and the path to the checkpoint as second argument.
