Tensorflow Neural Network for Regression

Question

I am using tensor flow library to build a pretty simple 2 layer artificial neural network to perform linear regression. My problem is that the results seem to be far from expected. I've been trying to spot my mistake for hours but no hope. I am new to tensor flow and neural networks so it could be a trivial mistake. Could anyone have an idea what i am doing wrong?

from __future__ import print_function

 import tensorflow as tf
 import numpy as np
 # Python optimisation variables
 learning_rate = 0.02

data_size=100000
data_length=100
train_input=10* np.random.rand(data_size,data_length);
train_label=train_input.sum(axis=1);
train_label=np.reshape(train_label,(data_size,1));

test_input= np.random.rand(data_size,data_length);
test_label=test_input.sum(axis=1);
test_label=np.reshape(test_label,(data_size,1));

x = tf.placeholder(tf.float32, [data_size, data_length])
y = tf.placeholder(tf.float32, [data_size, 1])

W1 = tf.Variable(tf.random_normal([data_length, 1], stddev=0.03), name='W1')
b1 = tf.Variable(tf.random_normal([data_size, 1]), name='b1')

y_ = tf.add(tf.matmul(x, W1), b1)


cost = tf.reduce_mean(tf.square(y-y_))                   
optimiser=tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
.minimize(cost)

init_op = tf.global_variables_initializer()

correct_prediction = tf.reduce_mean(tf.square(y-y_))    
accuracy = tf.cast(correct_prediction, tf.float32)


with tf.Session() as sess:
  sess.run(init_op)
  _, c = sess.run([optimiser, cost], 
                     feed_dict={x:train_input , y:train_label})
  k=sess.run(b1)
  print(k)                   
  print(sess.run(accuracy, feed_dict={x: test_input, y: test_label}))

Thanks for your help!

Which data are you training on? Have you tried comparing your RMSE with the RMSE obtained with a simpler linear regression (maybe use scikit-learn's implementation)? Have you tried varying the learning rate? Seems like you are only doing one pass (epoch) over your data: maybe you should train for several epochs? — valentin

Prasad Prasad · Accepted Answer · 2017-11-24T12:59:27

There are a number of changes you have to make in your code.

First of all, you have to perform training for number of epochs and also feed the optimizer training data in batches. Your learning rate was very high. Bias is supposed to be only one input for every dense (fully connected) layer. You can plot the cost (loss) value to see how your network is converging. In order to feed data in batches, I have made the changes in placeholders also. Check the full modified code:

from __future__ import print_function

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
# Python optimisation variables
learning_rate = 0.001  

data_size=1000  # Had to change these value to fit in my memory
data_length=10
train_input=10* np.random.rand(data_size,data_length);
train_label=train_input.sum(axis=1);
train_label=np.reshape(train_label,(data_size,1));

test_input= np.random.rand(data_size,data_length);
test_label=test_input.sum(axis=1);
test_label=np.reshape(test_label,(data_size,1));

tf.reset_default_graph()
x = tf.placeholder(tf.float32, [None, data_length])
y = tf.placeholder(tf.float32, [None, 1])

W1 = tf.Variable(tf.random_normal([data_length, 1], stddev=0.03), name='W1')
b1 = tf.Variable(tf.random_normal([1, 1]), name='b1')

y_ = tf.add(tf.matmul(x, W1), b1)


cost = tf.reduce_mean(tf.square(y-y_))                   
optimiser=tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cost)

init_op = tf.global_variables_initializer()

EPOCHS = 500
BATCH_SIZE = 32
with tf.Session() as sess:
    sess.run(init_op)

    loss_history = []
    for epoch_no in range(EPOCHS):
        for offset in range(0, data_size, BATCH_SIZE):
            batch_x = train_input[offset: offset + BATCH_SIZE]
            batch_y = train_label[offset: offset + BATCH_SIZE]

            _, c = sess.run([optimiser, cost], 
                     feed_dict={x:batch_x , y:batch_y})
            loss_history.append(c)


    plt.plot(range(len(loss_history)), loss_history)
    plt.show()

    # For running test dataset
    results, test_cost = sess.run([y_, cost], feed_dict={x: test_input, y: test_label})
    print('test cost: {:.3f}'.format(test_cost))
    for t1, t2 in zip(results, test_label):
        print('Prediction: {:.3f}, actual: {:.3f}'.format(t1[0], t2[0]))

Tensorflow Neural Network for Regression

1 Answers