Sakana AI's PC-ALM trains 1,000-layer networks without backpropagation
The new local-learning method uses augmented Lagrangians to match backpropagation performance in deep residual MLPs.
Researchers at Sakana AI have developed Augmented Lagrangian Predictive Coding (PC-ALM), a local-learning alternative to backpropagation capable of training residual multilayer perceptrons (MLPs) up to 1,000 layers deep. This development suggests a viable path toward training massive neural networks without the requirement of a global backward pass.
Standard backpropagation requires a strict sequential flow of data forward and then backward through the entire network. PC-ALM replaces these global passes with feedback control dynamical systems embedded in each layer. The method utilizes dual ascent on the augmented Lagrangian to drive these dynamics, enabling "ballistic" credit propagation. This process is significantly faster and more efficient than the slow, diffusive propagation found in traditional predictive coding. In linear networks, researchers found that PC-ALM converges to an equilibrium that produces exact backpropagation gradients.
The challenge of biological plausibility
Modern deep learning depends almost entirely on backpropagation, yet the process is widely viewed as biologically implausible. This is primarily due to "phase locking," the requirement that a network must complete a full forward pass before a backward pass can begin. While Predictive Coding (PC) has long been proposed as a more brain-like, local alternative, it has historically struggled with credit assignment. In very deep or narrow networks, traditional PC often fails to distribute supervision credit effectively, leading to a significant performance gap when compared to backpropagation.
Implications for hardware and neuroscience
If a local-learning rule can match backpropagation performance at scale, the implications for computing hardware are substantial. Moving away from global backward passes could enable more energy-efficient neuromorphic computing systems that process information in a truly parallel, decentralized manner. Beyond hardware, the success of PC-ALM provides a theoretical framework for understanding how the human brain might solve the multilayer credit assignment problem without relying on a global synchronization mechanism.
Future directions
While the researchers demonstrated that PC-ALM can handle residual MLPs up to 1,000 layers, further testing is required to see how the method scales to other architectures. The team noted that in nonlinear networks up to 128 layers deep, PC-ALM matches backpropagation performance across all width-depth regimes. The next phase of research will likely focus on whether these local dynamics can be applied to transformer-based architectures or other complex models used in generative AI.