PC-ALM Trains 1000-Layer Networks Without Backpropagation
Backprop Alternative: Augmented Lagrangian Predictive Coding
PC-ALM, a local alternative to backpropagation, trains residual MLPs up to 1000 layers by equipping each layer with a feedback control dynamical system. It extends predictive coding by adding dual neurons (Lagrange multipliers) per layer, making each layer a PI feedback controller. In linear networks, these dual neurons converge to exact backprop credit signals using only local computation. Experiments show PC-ALM overcomes predictive coding's signal decay problem and nearly matches backprop's performance.
In the limiting case of linear networks, the dual neurons converge to the exact backprop credit signals, despite using only local computation.
- lukeinator42
There is a lot of interesting research into predictive coding as an alternative means to solve the credit assignment problem that might be a more plausible model of what happens in the brain.
I really liked this paper that showed using a predictive coding learning rule leads to the exact same gradients as backprop in arbitrary networks:
Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs https://direct.mit.edu/neco/article/34/6/1329/110646/Predict...
- txhwind
Nice introduction to a simple but useful idea! The Lagrangian works like a time-smoothed optimizing direction state, but it can be placed on any wire, even at non-differentiable boundary! Can it be better than existing training methods for discrete components like argmax, MoE or VQ-VAE? Maybe networks can be composed by a lot of learnable discrete components, or even bits and gates finally.
- Jeff_Brown
Could this relate to continual learning? It lets you update without pausing the entire system.
- rao-v
I wonder if you could take a traditional backprop trained LLM and apply this approach to finetuning it (presumably needs less memory and compute?). It could be another entry in the spectrum between LORA and full fine tuning.
- AIorNot
Oh wow the theoretical implications in neuroscience exite me here - is this a potential model of Fristons Markov Blanket concept
“ Probably the most ambitious and all-encompassing version of the ‘Bayesian turn’ in cognitive science is
the free energy principle (FEP). The FEP is a mathematical framework, developed by Karl Friston and
colleagues (Friston, Kilner, and Harrison 2006; Friston et al. 2010; Friston 2010; Friston et al. 2017a;
Friston 2019), which specifies an objective function that any self-organizing system needs to minimize in
order to ensure adaptive exchanges with its environment. One major appeal of the FEP is that it aims for
(and seems to deliver) an unprecedented integration of the life sciences (including psychology,
neuroscience, and theoretical biology). The difference between the FEP and earlier inferential theories
(e.g., Gregory 1980, Grossberg 1980, Rao and Ballard 1999, Lee and Mumford 2003) is that not only
perceptual processes, but also other cognitive functions such as learning, attention, and action planning
can be subsumed under one single principle: the minimization of free energy through the process of active
inference (Friston 2010; Friston et al. 2017). ”