When should you update weights in neural network using backpropagation?

Question

Let's say I have a 3 layer fully-connected neural network. I am implementing backpropagation algorithm. My question is, should I first calculate deltas and then after backpropagation is done, update the weights, or should I do it as I backpropagate through layers? I have seen both ways in internet tutorials.

I'm not sure because if I update weights during backpropagation I use newly updated weights (hidden to outputs weights) to calculate hidden layer deltas and I'm not sure is this is desired.

Sorry if I used incorrect terminology, I am new to this and trying to learn.

Answer 1

The classical approach is to update all the weights simultaneously, as a single operation. This may lead to the so called covariance shift (last layers are updated assuming old weights from the early layers), but that is where batch normalization helps.