Weights in Neural Network

Question

I am reading : https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6

I saw following code:

import numpy as np

def sigmoid(x):
    return 1.0/(1+ np.exp(-x))

def sigmoid_derivative(x):
    return x * (1.0 - x)

class NeuralNetwork:
    def __init__(self, x, y):
        self.input      = x
        self.weights1   = np.random.rand(self.input.shape[1],4) 
        self.weights2   = np.random.rand(4,1)                 
        self.y          = y
        self.output     = np.zeros(self.y.shape)

    def feedforward(self):
        self.layer1 = sigmoid(np.dot(self.input, self.weights1))
        self.output = sigmoid(np.dot(self.layer1, self.weights2))

    def backprop(self):
        # application of the chain rule to find derivative of the loss function with respect to weights2 and weights1
        d_weights2 = np.dot(self.layer1.T, (2*(self.y - self.output) * sigmoid_derivative(self.output)))
        d_weights1 = np.dot(self.input.T,  (np.dot(2*(self.y - self.output) * sigmoid_derivative(self.output), self.weights2.T) * sigmoid_derivative(self.layer1)))

        # update the weights with the derivative (slope) of the loss function
        self.weights1 += d_weights1
        self.weights2 += d_weights2


if __name__ == "__main__":
    X = np.array([[0,0,1],
                  [0,1,1],
                  [1,0,1],
                  [1,1,1]])
    y = np.array([[0],[1],[1],[0]])
    nn = NeuralNetwork(X,y)

    for i in range(1500):
        nn.feedforward()
        nn.backprop()

    print(nn.output)

Shouldn't the weights be a 4x4 random matrix because we have 4 neurons in hidden layers and 4 input values so the total number of weight should be 16 but the following code assigns a matrix of 2x4 in the init function and creates a dot product?

there is only one hidden layer and one output layer in this network, not four. — Marat

Mustafa Aydın Mustafa Aydın · Accepted Answer · 2020-05-06T22:04:07

Your input matrix X suggests that number of samples is 4 and the number of features is 3. The number of neurons in the input layer of a neural network is equal to the number of features*, not number of samples. For example, consider that you have 4 cars and you chose 3 features for each of them: color, number of seats and origin country. For each car sample, you feed these 3 features to the network and train your model. Even if you have 4000 samples, the number of input neurons do not change; it's 3.

So self.weights1 is of shape (3, 4) where 3 is number of features and 4 is the number of hidden neurons (this 4 has nothing to do with the number of samples), as expected.

*: Sometimes the inputs are augmented by 1 (or -1) to account for bias, so number of input neurons would be num_features + 1 in that case; but it's a choice of whether to deal with bias seperately or not.

Weights in Neural Network

1 Answers