Are there alternatives to backpropagation?

6
votes

I know a neural network can be trained using gradient descent and I understand how it works.

Recently, I stumbled upon other training algorithms: conjugate gradient and quasi-Newton algorithms. I tried to understand how they work but the only good intuition I could get is that they use higher order derivative.

My questions are the following: are those alternative algorithms I mentioned fundamentally different from a backpropagation process where weights are adjusted by using the gradient of the loss function? If not, is there an algorithm to train a neural network that is fundamentally different from the mechanism of backpropagation?

Thanks

2
neural-networkbackpropagationgradient-descent
Imho backpropagation is not a learning algorithm. Its a gradient calculation algorithm. Learning is usually done by stochastic gradient then. But you could also do bfgs and co. Of course you could also adjust weights by genetic algorithms and such, without real gradients - sascha

2 Answers

5
votes

Conjugate gradient and quasi-Newton algorithms are still gradient descent algorithms. Backpropagation (or backprop) is nothing more than a fancy name to a gradient computation.

However, the original question of alternatives to backprop is very important. One of the recent alternatives, for example, is equilibrium propagation (or shortly eqprop).

1
votes

Neuroevolution of augmenting topologies or NEAT is another way to learn the topology of the network and weights/biases of the network using the genetic algorithm.