Introducing PC-ALM, a local-learning alternative to backpropagation.
- Source
- Sakana AI
- Date
Introducing PC-ALM, a local-learning alternative to backpropagation. Our method trains 1000-layer neural nets using only local dynamics, and without backprop. Blog: https://t.co/bBGCgalqKW Standard deep learning relies on backpropagation. The brain, however, cannot implement backpropagation, at least not exactly. How can a physical system, such as the brain, solve multilayer credit assignment without explicit use of backprop? We look for inspiration in two related fields: distributed optimization and NeuroAI. In NeuroAI, predictive coding asks each neuron activation to solve an energy-based inference problem instead of using a standard forward pass. That inference step can be implemented as energy-minimization dynamics on local prediction errors. This perspective -- each layer as a dynamical system -- has proven promising, but performance of predictive coding hasn't scaled well with depth. Credit signals at far ends of the network struggle to diffuse into internal layers. We turn to distributed optimization, generalizing predictive coding to use an augmented Lagrangian instead of energy. This motivation stems back to a classic 1988 paper by LeCun, showing that the Lagrange…
Context
Sakana AI says the brain cannot implement backpropagation, the standard method that sends error signals backward through a whole network, at least not exactly. Its PC-ALM is a local-learning alternative in which each layer updates using only local dynamics. It builds on predictive coding, where layers reduce local prediction errors, an approach that has scaled poorly with depth because credit signals from the far end struggle to reach internal layers.
PC-ALM adds a Lagrange multiplier to each layer, an accumulated record of that layer's prediction error, following a 1988 LeCun result that identifies such multipliers with gradients of a supervised loss. The paper's abstract reports matching backpropagation in nonlinear networks up to depth 128, notably in deep narrow networks where predictive coding underperforms. Sakana's post also claims 1000-layer training, which the abstract does not report, and neither the post nor the abstract gives accuracy on a standard task.
Checking sign-in…
Loading comments…