Neural Network loss starts increasing while acc is increasing on both train/val data sets

Question

For past few days I have been debugging my NN but I can't find an issue.

I've created total raw implementation of multi-layer perceptron for identifying MNIST dataset images.

Network seems to learn because after train cycle test data accuracy is above 94% accuracy. I have problem with loss function - it starts increasing after a while, when test/val accuracy reaches ~76%.

Can someone please check my forward/backprop math and tell me if my loss function is properly implemented, or suggest what might be wrong?

NN structure:

• input layer: 758 nodes, (1 node per pixel)
• hidden layer 1: 300 nodes
• hidden layer 2: 75 nodes
• output layer: 10 nodes

NN activation functions:

• input layer -> hidden layer 1: ReLU
• hidden layer 1 -> hidden layer 2: ReLU
• hidden layer 2 -> output layer 3: Softmax

NN Loss function:

• Categorial Cross-Entropy

Full CLEAN code available here as Jupyter Notebook.

Neural Network forward/backward pass:

def train(self, features, targets):
        n_records = features.shape[0]

        # placeholders for weights and biases change values
        delta_weights_i_h1 = np.zeros(self.weights_i_to_h1.shape)
        delta_weights_h1_h2 = np.zeros(self.weights_h1_to_h2.shape)
        delta_weights_h2_o = np.zeros(self.weights_h2_to_o.shape)
        delta_bias_i_h1 = np.zeros(self.bias_i_to_h1.shape)
        delta_bias_h1_h2 = np.zeros(self.bias_h1_to_h2.shape)
        delta_bias_h2_o = np.zeros(self.bias_h2_to_o.shape)

        for X, y in zip(features, targets):
            ### forward pass
            # input to hidden 1
            inputs_to_h1_layer = np.dot(X, self.weights_i_to_h1) + self.bias_i_to_h1
            inputs_to_h1_layer_activated = self.activation_ReLU(inputs_to_h1_layer)

            # hidden 1 to hidden 2
            h1_to_h2_layer = np.dot(inputs_to_h1_layer_activated, self.weights_h1_to_h2) + self.bias_h1_to_h2
            h1_to_h2_layer_activated = self.activation_ReLU(h1_to_h2_layer)

            # hidden 2 to output
            h2_to_output_layer = np.dot(h1_to_h2_layer_activated, self.weights_h2_to_o) + self.bias_h2_to_o
            h2_to_output_layer_activated = self.softmax(h2_to_output_layer)

            # output
            final_outputs = h2_to_output_layer_activated 

            ### backpropagation
            # output to hidden2
            error = y - final_outputs
            output_error_term = error.dot(self.dsoftmax(h2_to_output_layer_activated))

            h2_error = np.dot(output_error_term, self.weights_h2_to_o.T)
            h2_error_term = h2_error * self.activation_dReLU(h1_to_h2_layer_activated)

            # hidden2 to hidden1
            h1_error = np.dot(h2_error_term, self.weights_h1_to_h2.T) 
            h1_error_term = h1_error * self.activation_dReLU(inputs_to_h1_layer_activated)

            # weight & bias step (input to hidden)
            delta_weights_i_h1 += h1_error_term * X[:, None]
            delta_bias_i_h1 = np.sum(h1_error_term, axis=0)

            # weight & bias step (hidden1 to hidden2)
            delta_weights_h1_h2 += h2_error_term * inputs_to_h1_layer_activated[:, None]
            delta_bias_h1_h2 = np.sum(h2_error_term, axis=0)

            # weight & bias step (hidden2 to output)
            delta_weights_h2_o += output_error_term * h1_to_h2_layer_activated[:, None]
            delta_bias_h2_o = np.sum(output_error_term, axis=0)

        # update the weights and biases     
        self.weights_i_to_h1 += self.lr * delta_weights_i_h1 / n_records
        self.weights_h1_to_h2 += self.lr * delta_weights_h1_h2 / n_records
        self.weights_h2_to_o += self.lr * delta_weights_h2_o / n_records
        self.bias_i_to_h1 += self.lr * delta_bias_i_h1 / n_records
        self.bias_h1_to_h2 += self.lr * delta_bias_h1_h2 / n_records
        self.bias_h2_to_o += self.lr * delta_bias_h2_o / n_records

Activation function implementation:

def activation_ReLU(self, x):
    return x * (x > 0)

def activation_dReLU(self, x):
    return 1. * (x > 0)

def softmax(self, x):
    z = x - np.max(x)
    return np.exp(z) / np.sum(np.exp(z))

def dsoftmax(self, x):
    # TODO: vectorise math
    vec_len = len(x)
    J = np.zeros((vec_len, vec_len))
    for i in range(vec_len):
        for j in range(vec_len):
            if i == j:
                J[i][j] = x[i] * (1 - x[j])
            else:
                J[i][j] = -x[i] * x[j]
    return J

Loss function implementation:

def categorical_cross_entropy(pred, target): 
    return (1/len(pred)) * -np.sum(target * np.log(pred))

Answer 1

I managed to find the problem.

Neural Network is large so I couldn't stick everything to this question. Though if you check my Jupiter Notebook you could see implementation of my Softmax activation function and how do I use it in train cycle.

Problem with Loss miscalculation was caused by the fact my Softmax implementation worked only for ndarray dim == 1.

During training step I have put only ndarray with dim 1 to activtion function so NN learned well, but my run() function was returning wrong predictions as I have inserted whole test data to it, not only single row of it in for loop. Because of that it calculated Softmax "matrix-wise" rather than "row-wise".

This is very fast fix for it:

   def softmax(self, x):
        # TODO: vectorise math to speed up computation
        softmax_result = None
        if x.ndim == 1:
            z = x - np.max(x)
            softmax_result = np.exp(z) / np.sum(np.exp(z))
            return softmax_result
        else:
            softmax_result = []
            for row in x:
                z = row - np.max(row)
                row_softmax_result = np.exp(z) / np.sum(np.exp(z))
                softmax_result.append(row_softmax_result)
            return np.array(softmax_result)

Yet this code should be vectorised to avoid for loops and ifs if possible because currently it's ugly and takes too much PC resources.