I started a very simple RNN project to solidify my knowledge in TF, basically a simple sequence generator using LSTMs and TF. The project is just a many-to-one sequence generation, the input is a 4 integer window and the output has only one float for each window. The minimum number of the input is 1 and the maximum is 61, so I can predict from 61 and forward. I just used a batch of all inputs, which has shape [58,4,1] and the output with shape [58,1]. For better visualization, the inputs and outputs have been written below.
Inputs Labels
[[[ 1],[ 2],[ 3],[ 4]], -------> [[0.0493],
[[ 2],[ 3],[ 4],[ 5]], -------> [0.0634],
[[ 3],[ 4],[ 5],[ 6]], -------> [0.0773],
[[ 4],[ 5],[ 6],[ 7]], -------> [0.0909],
.. .. .. .. -------> ... ,
[[55],[56],[57],[58]], -------> [0.5503],
[[56],[57],[58],[59]], -------> [0.5567],
[[57],[58],[59],[60]], -------> [0.5630],
[[58],[59],[60],[61]]] -------> [0.5693]]
The training part went very well and I could achieve something around 0.991 accuracy with 500 epochs, but when I try to predict some values from 61 to 118, the output has a fixed step down for all predicted values but somehow has the right behavior.
Because the purpose of this project is just for learning the basics, I decided to use the simplest functions in TF, so the seq2seq facilities have been left off. The code for the RNN is written below
def build_lstm(cell_lengh, cell_depth, batch_size, keep_prob):
def lstm_row(cell_length, keep_prob):
cell_row = tf.contrib.rnn.BasicLSTMCell(cell_lengh)
cell_row = tf.contrib.rnn.DropoutWrapper(cell_row, keep_prob)
return cell_row
cell = tf.contrib.rnn.MultiRNNCell([lstm_row(cell_lengh, keep_prob) for _ in range(cell_depth)])
initial_state = cell.zero_state(batch_size, tf.float32)
return cell, initial_state
tf.reset_default_graph()
inputs = tf.placeholder(tf.float32, [None, feature_length, 1], name='inputs')
labels = tf.placeholder(tf.float32, [None, output_length], name='labels')
keep_prob = tf.placeholder(tf.float32, name='kpprob')
lstm_cell, initial_state = build_lstm(40, 2, batch_size=batch_size, keep_prob=keep_prob)
lstm_output, final_state = tf.nn.dynamic_rnn(lstm_cell, inputs, initial_state=initial_state)
lstm_outout_seq = lstm_output[:,-1,:]
dense_0 = tf.layers.dense(inputs=lstm_outout_seq, units=120, activation=tf.nn.relu)
dropout_0 = tf.layers.dropout(dense_0, rate=0.7)
with tf.variable_scope('sigmoid'):
W = tf.Variable(tf.truncated_normal((120, 1), stddev=0.1), name='weights')
b = tf.Variable(tf.zeros(1), name='bias')
logits = tf.matmul(dropout_0, W) + b
output = tf.nn.sigmoid(logits, name='output')
loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=labels))
correct_predictions = tf.abs(output - labels)
total_correct = tf.ones_like(correct_predictions)
accuracy = tf.reduce_mean(total_correct - correct_predictions)
learning_rate = tf.placeholder(tf.float32, name='learning_rate')
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)
l_rate = 0.001
epochs = 500
kp_prob = 0.7
with tf.Session() as session:
session.run(tf.global_variables_initializer())
for e in range(epochs):
new_state = session.run([initial_state])
feeder = {
inputs: wnd_x,
labels: wnd_y_scl,
keep_prob: kp_prob,
learning_rate: l_rate,
initial_state: new_state
}
session_loss,
session_accuracy,
session_output, _,
last_state = session.run([loss, accuracy, output,
optimizer, final_state], feed_dict=feeder)
print('Epoch {0}/{1}:\t'.format(e, epochs),
'training loss {0}\t'.format(session_loss),
'accuracy {0}\t'.format(session_accuracy))
new_state = session.run([initial_state])
feeder = {
inputs: unseen_data_rsp,
keep_prob: 1.0,
initial_state: new_state
}
session_output = session.run([output], feed_dict=feeder)
As mentioned before, during the inference phase, the predictions have fixed step down but somehow have the right behavior, i.e. the derivate of the curves changes correctly for the time-steps.
During the training phase I have the following output:
Epoch 999/1000: training loss = 0.5913468599319458 | accuracy = 0.9909629225730896
Input Label Output
[[ 1],[ 2],[ 3],[ 4]] --> [0.0493] ... [0.0591]
[[ 2],[ 3],[ 4],[ 5]] --> [0.0634] ... [0.0802]
[[ 3],[ 4],[ 5],[ 6]] --> [0.0773] ... [0.0777]
[[ 4],[ 5],[ 6],[ 7]] --> [0.0909] ... [0.1035]
.. .. .. .. ... ... ...
[[55],[56],[57],[58]] --> [0.5503] ... [0.5609]
[[56],[57],[58],[59]] --> [0.5567] ... [0.5465]
[[57],[58],[59],[60]] --> [0.5630] ... [0.5543]
[[58],[59],[60],[61]] --> [0.5693] ... [0.5614]
And during inference phase I have the following output:
Input Prediction
[[ 58],[ 59],[ 60],[ 61]] --> [0.4408]
[[ 59],[ 60],[ 61],[ 62]] --> [0.4459]
[[ 60],[ 61],[ 62],[ 63]] --> [0.4510]
[[ 61],[ 62],[ 63],[ 64]] --> [0.4559]
... ... ... ... ... ...
[[112],[113],[114],[115]] --> [0.6089]
[[113],[114],[115],[116]] --> [0.6101]
[[114],[115],[116],[117]] --> [0.6113]
[[115],[116],[117],[118]] --> [0.6124]
As you can see, the first input of the inference is the same of the last input of the training. What I don't understand here is why the same input gave me 2 different outputs and why theses output has a fixed step down, around 0.11. Thank you guys for any help and sorry for the long text, I can make it shorter upon request.
