Neural Network Backpropagation

Question

I am trying to understand the error backpropagation algorithm to be used in Artificial Neural Networks.

As far as I understand, each neuron has as an output the result of a sigmoid function, which takes as parameter the sum of the product of each weight/input pair plus a "bias" value.

Once the total error of the network is calculated, the derivative of the error with respect to the various weights can be used to find "local minimum" of the error function. These should be the weights where the error is minimum.

However, points where the derivative is zero could also in theory be local maximums... How can I solve this problem?

Cheers :)

Answer 1

Neural network itself couldn't free from the local minimum problem. So we need additional techniques.

One easy way to reduce the local minimum problem is using momentum. (but, this is not the holy grail) update weights using historical movement of delta (such as moving average) will be useful. E.g if in the certain epoch, delta was -1, but the recent delta history was +1, +2, +4, +2. And we defined the MA size 5. then the actual delta in this epoch will be (+1 +2 +4 +2 -1)/5 = +1.6

Maybe, if you use exponential decaying function to the moving window weight, this momentum approach shows more improved result.