Parcae shows that adding recurrence (looping a smaller model) can match a Transformer twice its size, offering a compute- and memory-efficient path to quality — useful if you're weighing model size against inference cost.
It provides the first scaling laws for looping, giving concrete guidance on when recurrence beats simply adding parameters or data.
Parcae is a stable looped language model that matches the quality of a Transformer twice its size — a 770M model reaching 1.3B-level performance.
We introduce the first scaling laws for looping and show that increasing recurrence, not just data, is a compute-efficient path to bet
Transcript
Parcae is a stable looped language model that matches the quality of a Transformer twice its size — a 770M model reaching 1.3B-level performance. We introduce the first scaling laws for looping and show that increasing recurrence, not just data, is a compute-efficient path to bet
