Tensorflow RNN many to many Time series for binary labels

Question

I am trying to train a basic recurrent neural network (many to many). The input data has a single feature (a sin function) and labels are binary: (1 and 2)

I am able to train it using a MSE loss function but I have some problems when I try to replace it for xentropy.

Here is the code I have so far that works for non-discrete labels:

import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np

n_steps = 20
n_inputs = 1
n_neurons = 100
n_outputs = 1
learning_rate = 0.001

# Create data 
fs = 1000 # sample rate 
f = 2 # the frequency of the signal
x = np.arange(fs) # the points on the x axis for plotting

# training features
dfX = np.array([ np.sin(2*np.pi*f * (i/fs)) for i in x]) 

#labels
dfX2 = np.array([ np.sin(2*np.pi*f * (i/fs)) for i in x]) 
dfX2[dfX2 < 0] = 1
dfX2[dfX2 > 0] = 2

# RNN

X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])
y = tf.placeholder(tf.float32, [None, n_steps, n_outputs])

cell = tf.contrib.rnn.OutputProjectionWrapper( tf.contrib.rnn.BasicRNNCell(num_units=n_neurons, activation=tf.nn.relu),output_size=n_outputs)
outputs, states = tf.nn.dynamic_rnn(cell, X, dtype=tf.float32)


loss = tf.reduce_mean(tf.square(outputs - y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
training_op = optimizer.minimize(loss)
init = tf.global_variables_initializer()

n_iterations = 1000
batch_size = 200
n_epouchs=10

with tf.Session() as sess:
  init.run()
  for epouch in range(n_epouchs):
    for iteration in range(n_iterations-200):
      X_batch= dfX[iteration:iteration+batch_size]
      X_batch= X_batch.reshape(-1,n_steps,n_inputs)
      y_batch= dfX2[(iteration+1):(iteration+batch_size+1)]
      y_batch= y_batch.reshape(-1,n_steps,n_outputs)
      sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
      if iteration % 100 == 0:
    mse = loss.eval(feed_dict={X: X_batch, y: y_batch})
    print(iteration,"--",epouch, "\tMSE:", mse)
  X_new1= dfX[37:37+batch_size]
  X_new1= X_new1.reshape(-1,n_steps,n_inputs)
  y_pred1 = sess.run(outputs, feed_dict={X: X_new1})

I am trying to change the loss function to something like this ( as i want to predict discrete labels):

xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=outputs)
loss = tf.reduce_mean(xentropy)

But It does not work

vijay m vijay m · Accepted Answer · 2018-05-14T14:50:34

The code has to have the following changes to work for cross_entropy_loss:

1: One hot labels: They should take 0 or 1. So change your code to:

dfX2[dfX2 < 0] = 0
dfX2[dfX2 > 0] = 1

2: For 2 class problem the network output layer should be 2, so your RNN should be:

cell = tf.contrib.rnn.OutputProjectionWrapper( tf.contrib.rnn.BasicRNNCell(
          num_units=n_neurons, activation=tf.nn.relu),
        output_size=2) #Output size is set to 2.

3: Since the inputs are not one-hot coded, you need to convert them to one-hot for cross entropy loss:

 xentropy =  tf.nn.softmax_cross_entropy_with_logits_v2(
               labels=tf.one_hot(tf.cast(y, tf.int32),2), logits=outputs)

Making the above changes will get similar MSE scores:

0 -- 0  MSE: 0.60017127
100 -- 0    MSE: 0.13623504
200 -- 0    MSE: 0.07625882
300 -- 0    MSE: 0.006987947

Tensorflow RNN many to many Time series for binary labels

1 Answers