TensorFlow neural network wired incorrectly, but what is missing?

Question

I have created a very simple TensorFlow neural network, but clearly I must have skipped a step somewhere or mixed up sample code from different tutorials, because the results are nonsensical, and the training error only increases with each epoch.

Here's a fully self-contained example (MVCE), trying to train the network to calculate the square function:

import tensorflow as tf
import numpy as np

# hard-coded input and labels for demonstration
training_x = np.array([[1.], [2.],[3.],[4.],[5.]]).T
labels_training = np.array([[1.],[4.],[9.],[16.],[25.]]).T

# Hyperparameters
num_epochs = 1000
learning_rate = 0.001
LAYERS = 3

# setup the Neural Network
INPUT = len(training_x)
OUTPUT = len(labels_training)
X = tf.placeholder(tf.float32, shape=[INPUT,None])
Y = tf.placeholder(tf.float32, shape=[OUTPUT, None])
parameters = {
    'W1': tf.Variable(np.random.randn(LAYERS,INPUT), dtype=tf.float32),
    'b1': tf.Variable(np.zeros([LAYERS,1]), dtype=tf.float32),
    'W2': tf.Variable(np.random.randn(OUTPUT,LAYERS), dtype=tf.float32),
    'b2': tf.Variable(np.zeros([OUTPUT,1]), dtype=tf.float32)
}
Z1 = tf.add(tf.matmul(parameters['W1'], X), parameters['b1']) # W1*X + b
A2 = tf.nn.relu(Z1)
Z2 = tf.add(tf.matmul(parameters['W2'], A2), parameters['b2']) 
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=Z2,  labels=Y)) 
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)
    for epoch in range(num_epochs):
        _ , c = sess.run([optimizer, cost], feed_dict={X: training_x, Y: labels_training}) 

        if epoch % 200 == 0:
            print ("Cost after epoch %i: %f" % (epoch, c))

    # Test predictions by computing the output using training set as input 
    output = sess.run(Z2, feed_dict={X: training_x})
    print(np.array2string(output, precision=3))

Example output (YMMV due to the random initialization vector):

Cost after epoch 0: 158.512558
Cost after epoch 200: 227.178513
Cost after epoch 400: 319.617218
Cost after epoch 600: 436.471069
Cost after epoch 800: 577.651733
[[23.437 38.291 53.145 67.999 82.852]]

Paprikadobi Paprikadobi · Accepted Answer · 2018-07-23T19:34:10

I tried your code and I think you should change cost function. If I change it to cost = tf.reduce_mean(tf.losses.mean_squared_error(labels = Y, predictions = Z2)) then it works better.

EDIT: And when I didn't transpose your input and output data it reduces cost to 0 in under 200 epochs.

TensorFlow neural network wired incorrectly, but what is missing?

2 Answers