I am getting an accuracy of 1.0 every time in neural network

Question

I am doing binary classification using multilayer perceptron with numpy and tensorflow.

input matrix is of shape = (9578,18) labels has the shape = (9578,1)

Here's the code:

#preprocessing
input = np.loadtxt("input.csv", delimiter=",", ndmin=2).astype(np.float32)
labels = np.loadtxt("label.csv", delimiter=",", ndmin=2).astype(np.float32)

train_size = 0.9

train_cnt = floor(inp.shape[0] * train_size)
x_train = input[0:train_cnt]
y_train = labels[0:train_cnt]
x_test = input[train_cnt:]
y_test = labels[train_cnt:]

#defining parameters

learning_rate = 0.01
training_epochs = 100
batch_size = 50
n_classes = labels.shape[1]
n_samples = 9578
n_inputs = input.shape[1]
n_hidden_1 = 20
n_hidden_2 = 20

def multilayer_network(X,weights,biases,keep_prob):
'''
X: Placeholder for data inputs
weights: dictionary of weights
biases: dictionary of bias values

'''
#first hidden layer with sigmoid activation
# sigmoid(X*W+b)
layer_1 = tf.add(tf.matmul(X,weights['h1']),biases['h1'])
layer_1 = tf.nn.sigmoid(layer_1)
layer_1 = tf.nn.dropout(layer_1,keep_prob)

#second hidden layer
layer_2 = tf.add(tf.matmul(layer_1,weights['h2']),biases['h2'])
layer_2 = tf.nn.sigmoid(layer_2)
layer_2 = tf.nn.dropout(layer_2,keep_prob)

#output layer
out_layer = tf.matmul(layer_2,weights['out']) + biases['out']

return out_layer

#defining the weights and biases dictionary

 weights = {
'h1': tf.Variable(tf.random_normal([n_inputs,n_hidden_1])),
'h2': tf.Variable(tf.random_normal([n_hidden_1,n_hidden_2])),
'out': tf.Variable(tf.random_normal([n_hidden_2,n_classes]))
 }

 biases = {
'h1': tf.Variable(tf.random_normal([n_hidden_1])),
'h2': tf.Variable(tf.random_normal([n_hidden_2])),
'out': tf.Variable(tf.random_normal([n_classes]))
 }
keep_prob = tf.placeholder("float")


X = tf.placeholder(tf.float32,[None,n_inputs])
Y = tf.placeholder(tf.float32,[None,n_classes])

predictions = multilayer_network(X,weights,biases,keep_prob)

#cost function(loss) and optimizer function
cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=predictions,labels=Y))

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

#running the session
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)


#for loop

for epoch in range(training_epochs):
    avg_cost = 0.0
    total_batch = int(len(x_train) / batch_size)
    x_batches = np.array_split(x_train, total_batch)
    y_batches = np.array_split(y_train, total_batch)
    for i in range(total_batch):
        batch_x, batch_y = x_batches[i], y_batches[i]
        _, c = sess.run([optimizer, cost], 
                        feed_dict={
                            X: batch_x, 
                            Y: batch_y, 
                            keep_prob: 0.8
                        })
        avg_cost += c / total_batch

        print("Epoch:", '%04d' % (epoch+1), "cost=", \
            "{:.9f}".format(avg_cost))

print("Model has completed {} epochs of training".format(training_epochs))
correct_prediction = tf.equal(tf.argmax(predictions, 1), tf.argmax(Y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
print("Accuracy:", accuracy.eval({X: x_test, Y: y_test,keep_probs=1.0}))

After running my model for 100 epochs, the cost decreases after each epoch which means that the network is working okay, but the accuracy is coming out to be 1.0 everytime and I have no clue why as I am sort of a beginner when it comes to neural networks and how they function. So any help will be much appreciated. Thanks!

Edit: I tried checking the predictions matrix after each epoch and I am getting all zeros in that, everytime. I used the following code in my for loop with epochs to check the predictions matrix:

    for epoch in range(training_epochs):
    avg_cost = 0.0
    total_batch = int(len(x_train) / batch_size)
    x_batches = np.array_split(x_train, total_batch)
    y_batches = np.array_split(y_train, total_batch)
    for i in range(total_batch):
        batch_x, batch_y = x_batches[i], y_batches[i]
        _, c,p = sess.run([optimizer, cost,predictions], 
                        feed_dict={
                            X: batch_x, 
                            Y: batch_y, 
                            keep_prob: 0.8
                        })
        avg_cost += c / total_batch

    print("Epoch:", '%04d' % (epoch+1), "cost=", \
            "{:.9f}".format(avg_cost))
    y_pred = sess.run(tf.argmax(predictions, 1), feed_dict={X: x_test,keep_prob:1.0})
    y_true = sess.run(tf.argmax(y_test, 1))
    acc = sess.run(accuracy, feed_dict={X: x_test, Y: y_test,keep_prob:1.0})
    print('Accuracy:', acc)
    print ('---------------')
    print(y_pred, y_true)

print("Model has completed {} epochs of training".format(training_epochs))

Here's output of 1 epoch:

Epoch: 0001 cost= 0.543714217
Accuracy: 1.0
---------------
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] [0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

Answer 1

You aren't calling sess.run on the predictions. That means that it is currently a variable representing the tensorflow graph, rather than the values predicted.

Replace your _, c = sess.run([optimizer, cost], ...) with _, c, p = sess.run([optimizer, cost, predictions], ...). Then do the correct_prediction calculations on the p values that you get. Similarly, the truth values are batch_y, since your Y variable is also a tensorflow graph object. As such, you're now going to be working in numpy variables, so the argmax call should be done with np instead of tf. I believe that should do the trick.

If you want to do it in tensorflow, move the correct prediction and accuracy computations up to where you compute cost, and change your sess.run line to: _, c, a = sess.run([optimizer, cost, accuracy], ...)

For some explanation as to why you were getting 100%, you had the line correct_prediction = tf.equal(tf.argmax(predictions, 1), tf.argmax(Y, 1)), where both predictions and Y are tensorflow graph variables. You can think of those as wrappers for where the values will be flowing through when you call sess.run(). So when you print the accuracy, you're comparing tensorflow graph operations to tensorflow graph operations, and I guess the backend treats them as always being equal.

EDIT: Sample code for the two different mentioned methods below. Not 100% sure it works since I can't test it easily (I don't have your data), but it should be something like this.

First method:

    _, c, p = sess.run([optimizer, cost, predictions], ...)
    .
    .
    .
correct_prediction = np.equal(np.argmax(p, axis=1), np.argmax(batch_y, axis=1))
accuracy = np.mean(correct_prediction)

Second method:

cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=predictions,labels=Y))
correct_prediction = tf.equal(tf.argmax(predictions, 1), tf.argmax(Y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
.
.
.
    for i in range(total_batch):
        batch_x, batch_y = x_batches[i], y_batches[i]
        _, c, a = sess.run([optimizer, cost, accuracy], 
                        feed_dict={
                            X: batch_x, 
                            Y: batch_y, 
                            keep_prob: 0.8
                        })
        print(a)

EDIT 2: While the above information is still true, there's another problem as well. It doesn't make sense to use cross entropy and accuracy when there's only one class that you're predicting. If you call argmax on something of length 1, then you'll always get 0, since that's the only position that exists! Accuracy and cross entropy only make sense in the context of class-based predictions, where your truth values are a one-hot vector across the list of classes.

Answer 2

In your code I see x_train and x_test, so it looks like you've split your dataset a training/testing set correct? It's important that these two sets are independent of each other... i.e. the instances in each set are unique from each other. It's possible that you have a dataset with many repeating instances, if so... then your random split of the test set probably is just a subset of the training set and you are overtraining.

One useful tool to debug is a plot of the learning curve. To do this, simply plot the accuracy of the training and test set after every training epoch. It can tell you a lot about the model: under-fitting, over-fitting, stagnant learning, etc.