CNN model with Tensorflow

Question

I am doing character recognition by building a Convolutional Neural Network model in Tensorflow. My model has 2 Conv layer followed by 2 fully connected layer. I have around 78K images from training and 13K images for testing. When I execute the model, I got accuracy around 92.xx% on test set. When I visualized my accuracy and loss curve on Tensorboard. I got a vertical line, but I did not know why I am getting this? I got curve like this Accuracy and Cross Entropy curve when viewed on tensorboard.

Also distribution curve of weights and biases shows a vertical line Left side shows testing parameters (weights and bias) and right side shows training parameters on first conv layer

Any help in this regard is highly appreciated. !!

def conv_layer(input, size_in, size_out, name="conv"):
with tf.name_scope(name):
w = tf.Variable(tf.random_normal([5, 5, size_in, size_out], stddev=0.1), name="W")
b = tf.Variable(tf.constant(0.1, shape=[size_out]), name="B")
conv = tf.nn.conv2d(input, w, strides=[1, 1, 1, 1],padding="VALID")
act = tf.nn.relu(conv + b)
tf.summary.histogram("weights", w)
tf.summary.histogram("biases", b)
tf.summary.histogram("activations", act)
return tf.nn.max_pool(act, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding="SAME")`

def fc_layer(input, size_in, size_out, name="fc"):
with tf.name_scope(name):
w = tf.Variable(tf.random_normal([size_in, size_out], stddev=0.1), name="W")  # Truncated_normal
b = tf.Variable(tf.constant(0.1, shape=[size_out]), name="B")
act = tf.matmul(input, w) + b
tf.summary.histogram("weights", w)
tf.summary.histogram("biases", b)
tf.summary.histogram("activations", act)
return act

def model(use_two_conv, use_two_fc):
sess = tf.Session()
x = tf.placeholder(tf.float32, shape=[None, 1024], name="x")
x_image = tf.reshape(x, [-1, 32, 32, 1])
tf.summary.image('input', x_image, 3)
y = tf.placeholder(tf.float32, shape=[None,46], name="labels")

if use_two_conv:
  conv1 = conv_layer(x_image, 1, 4, "conv1")
  conv_out = conv_layer(conv1,4,16,"conv2")    
else:
  conv1 = conv_layer(x_image, 1, 16, "conv1")
  conv_out = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding="SAME")

flattened = tf.reshape(conv_out, [-1, 5 * 5 * 16])

if use_two_fc:
  fc1 = fc_layer(flattened, 5 * 5 * 16, 200, "fc1")
  relu = tf.nn.relu(fc1)
  tf.summary.histogram("fc1/relu", relu)
  logits = fc_layer(fc1, 200, 46, "fc2")        

else:
  logits = fc_layer(flattened, 5*5*16, 46, "fc")

I've run into this exact problem tons of times, but it's hard to diagnose a code problem with no code! Would you post your training code?. You probably wrote writer.add_summary(current_summary) instead of writer.add_summary(current_summary, epoch). If you post the code, I'll happily post this as an answer. — Dylan F
Thanks @DylanF. It worked. But I have one more problem, On this trained model, I want to test a new image but how should I restore variables defined in functions conv_layer() and fc_layer() because there are variables like "conv1/W:0", "conv2/W:0". How should I use this trained model for a new image? — Himanshu Tiwari
That's really a whole new question. Before you ask, though, you may want to reference this question. I use Tom's answer since it's the most up-to-date method. But the accepted answer is a valid solution too! — Dylan F

Dylan F Dylan F · Accepted Answer · 2017-12-30T18:39:10

When I've encountered this problem in the past, it's been a result of using

writer.add_summary(current_summary)

instead of

writer.add_summary(current_summary, epoch)

(using generic variable names because the relevant part of the asker's code isn't posted.) For instance,

summary_op = tf.summary.merge_all()
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    writer = tf.summary.FileWriter("/Whatever/Path", sess.graph)
    for iteration in range(1001):
        if epoch % 100 == 0:
            _, current_summary = sess.run([training_op, summary_op])
            writer.add_summary(current_summary, iteration)
        else:
            _ = sess.run(training_op)

CNN model with Tensorflow

1 Answers