What are hidden neurons doing in neural networks?

Question

So, example I have training set are 20 x 20 images or 400 features images to recognize written digits and I suppose that I have model 3 layers in NNs : input layer contain 400 neurons; one hidden layer contain 25 neurons and output layer contain 10 neurons. I read chapter 1 in neural network and deep learning.

At last part "A simple network to classify handwritten digits" (you can search with ctrl-f), it say that "neurons in the hidden layer detect whether or not the following images are present". So, Does it want to say neurons in hidden layer is present by image ? I think they are value pixels. If I'm wrong, how are images created from neurons in input layer ?

Answer 1

I'm sure you've heard the statement that hidden layers "extract higher order features" from the input features. In short, that is what they do by "examining" the relationships between these input features.

For a more in depth look at the question, let's use logistic regression.

Logistic regression is a similar type of learning algorithm, but with no hidden layers. In logistic regression (assuming binary classification for ease) you take the sigmoid function of the product of the input feature matrix and the weight matrix to output a value between 0 and 1 based on the network's certainty of the prediction .

So it takes input matrix multiplies by weight matrix and the applies the sigmoid function and that answer is the output layer. (i.e. sigmoid(input_matrix * weight_matrix) = output)

Well, in ANNs a hidden layer essentially stands between the above sigmoid function and the output layer. It takes the return of the sigmoid function and then multiplies that by another weight matrix for the set of neurons in that layer. This is done for each layer of neurons.

So for a network with one hidden layer it is - sigmoid(sigmoid(input_matrix * weight_matrix1) * weight_matrix2)

Now this might seem like it isn't that big of a difference, but it really comes in handy during back-propagation because we are able to determine each neuron's respective contribution to our loss function, and thus adjust all of the weight matrices accordingly, all the way back to the input layer's weights. This gives a much more sophisticated tool for modeling more complex functions.

So essentially, hidden layers aren't doing anything different from the input layer. They are just receiving some matrix, multiplying it by another and then computing the sigmoid function. Easy Peasy.