I am working with a multi-layer neural network. I intend to do mini-batch gradient descent. Suppose I have mini-batches of 100 over 1 million data points. I don't understand the part where I have to update the weights of the whole network. When I do a forward pass over these 100 samples, I sum all the errors over these 100 samples. What else do I do apart from this? Do I have to compute the hidden layer errors side by side too? When will they be computed?
1 Answers
5
votes
Batch learning in neural networks
You have to calculate the weight deltas for each neuron in all of the layers in you network, for each data instance in your (chosen) dataset. This is the same procedure as always in backpropagation.
Since you wish to use batch learning, you will delay the weight update step. Instead you should store these deltas and add up the deltas from the different instances in your batch after the batch has completed. Then use these newly manipulated deltas to update the weights in your network.
