Why Backpropagation Must Go Backward
Why back propagation goes backward
Backpropagation is often explained as propagating errors backward, but why not compute gradients forward? Gregory Gundersen shows that a forward pass would repeatedly propagate the same terms, leading to quadratic runtime. By passing local messages backward, backprop achieves linear time, solving a credit assignment problem where each node tells its upstream neighbors what they did wrong.
This process can be viewed as a solution to a kind of credit assignment problem: each node tells its upstream neighbors what they did wrong.
- omnicognate
I'm not familiar with the details of backprop in neural networks, but AIUI it's an application of automatic/algorithmic differentiation, which comes in two modes: forward and reverse.
Reverse mode is harder to implement as you need to retain state through the calculation, but it scales differently. Forward mode is O(number of inputs) while reverse is O(number of outputs). Seems obvious that reverse mode is what you want for training a neural network, where you have huge numbers of inputs and usually one output, the loss you're training on.
(And indeed that appears to be what the article is saying, in different language.)
- akssri
The intuition here is okay - but the math is hand-wavy with imprecise terms like "blow-up" etc.
The statements however, if taken to mean optimality, are also incorrect. Reverse-mode AD (backprop) is generally quite efficient for scalar outputs (more generally, when n_inputs >> n_outputs), but it's not strictly optimal even for this particular scalar-output case.
Consider for eg. a MLP, with 4-layers with dims (1, N, 1, N, 1) - reverse-mode here does ~3N multiplies, but the optimal is ~2N. The optimal ordering for gradient accumulation is in fact NP-hard on general DAGs, but such 'cross-mode' AD is apparently quite hard to implement and not often seen given the marginal gains.
Griewank-Walther's excellent book is a excellent reference for this and much more,
https://epubs.siam.org/doi/book/10.1137/1.9780898717761
They also had a library called ADOL-C that had mixed-mode.
- dkrylov
The real reason is that backprop is basically matrix multiplication and multiplying from left to right is way cheaper from right to left. Since on the left side you will have a scalar loss term and you keep vector - matrix multiplication through the network instead of doing matrix by matrix multiplication from the right side.