Parcae: Doing more with fewer parameters using stable looped models
- Source
- together.ai
- Author
- Together AI
- Date
Context
Parcae reuses the same language-model layers several times, adding computation without adding a separate set of weights for every pass. Repeated updates can become unstable during training, so the design constrains how state carries between loops and repeatedly supplies the original input representation. Together AI's research account describes this alongside additional training adjustments.
In the reported language-model experiments, a 770-million-parameter Parcae approached a 1.3-billion-parameter on the tested downstream aggregates, though its scores remained lower. This supports layer reuse as a route to better quality per stored parameter. The trade is additional repeated computation, and the result does not establish equal performance across all tasks.
- transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
“Parcae creates a new medium to scale quality by increasing recurrence rather than purely scaling data, opening up an efficient frontier for training memory-constrained on-device models.”
Together AI
“Our 770M Parcae matches the quality of a 1.3B parameter transformer trained on the same data, achieving the same performance with roughly half the parameters.”
Together AI
“Parcae achieves up to 6.3% lower validation perplexity than previous large-scale looped recipes.”
Together AI
“We establish the first scaling laws for looping , finding that compute-optimal training requires increasing looping and data in tandem .”
Together AI
“We personally found them to suffer from residual state explosion and loss spikes.”
Together AI
articleOne Model to Loop Them All: Elastic Latent Recurrence for Language ModelsFaisal Ladhak, Shantanu Acharya, Santiago Akle Serano, Simeng Sun, Boris Ginsburg, Krishna C Puvvada
articleTraining Compute-Optimal Large Language ModelsJordan Hoffmann et al.
articleScaling Laws for Neural Language ModelsJared Kaplan et al.
Checking sign-in…
Loading comments…