Training 1000-layer networks without backpropagation

• Sakana AI says standard predictive coding struggles with depth because credit signals at the far ends of a network fail to diffuse into internal layers, which is the scaling problem PC-ALM targets.
• PC-ALM adds dual neurons (Lagrange multipliers) to each layer's local dynamics, so every layer behaves like a PI feedback controller that drives its own prediction error down.
• The authors report PC-ALM propagates learning signals to seemingly arbitrary depth, with the clearest gains in deep, narrow networks where standard predictive coding struggles to learn.
• Sakana frames the work as a model of how brains might assign credit with only local coupling, and suggests it may suit neuromorphic hardware, where running dynamics is cheaper than on GPUs.

PC-ALM trains 1000-layer networks using only local error dynamics via an augmented-Lagrangian generalization of predictive coding, a concrete step toward biologically-plausible learning that scales past prior predictive-coding depth limits.

• articleSAIL: Scaling In-Context Imitation Learning
• articleIntroducing Sakana AI’s Frontier Intelligence Group (FIG)
• articleSakana Chatをアップデート：最新モデルに刷新、メモリー機能を追加
• articleSakana Marlinアップデート：レポートと対話する「Interactive Reading」と、出力スライドのPowerPoint対応

• postIntroducing PC-ALM, a local-learning alternative to backpropagation.Sakana AI
• postNetworks that obey Dale's principle learn without weight transportSakana AI
• podcastEric Jang – Building AlphaGo from scratchDwarkesh Patel

Checking sign-in…

Loading comments…