Efficient training of neural networks is difficult. Our second Connectionism post introduces Modular Manifolds, a theoretical step toward more stable and performant training by co-designing neural net optimizers with manifold constraints on weight matrices.
https://t.co/PGG4zy3u23
We explore a fundamental understanding of the geometry of neural network optimization.
A framing for treating training instability as a geometry problem, and how constraining weight matrices to manifolds changes what the optimizer should do.
