Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality
Source
arxiv.org
Author
Zeyuan Allen-Zhu
Date
Why it matters
Architecture gaps vanish at academic scale but show up at 1-2T tokens, and local token mixing (Canon layers) may explain much of what newer linear models add. That affects which architectures are worth trusting.
Key takeaways · AI-distilled
In academic-scale pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → (1.3B parameters, 100B tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →), carefully tuned architectures performed almost alike. The differences became measurable only at 1-2T tokens, across 64 from-scratch runs of up to 8B parameters at roughly $100k per 8B run.
Canon layers, which add local token mixing, saved about 30% of pretraining tokens for the same MMLU accuracy at 8B. With them, plain gated linear attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → matched Gated DeltaNet and beat Mamba2, so the author argues their extra machinery may mostly mix neighboring tokens.
Benchmark parity hid basic failures: linear models that matched Transformers on standard benchmarks could not reliably retrieve information from a 500-token context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →, and no model reliably beat random guessing on a short two-hop question, even at 8B/1T.
Small synthetic playgrounds under 100M parameters predicted these architectural differences. The author's practical advice is to put in-context reasoning examples into pretraining data early, rather than saving them for mid-training or post-training.
Terms in this piece · Glossary
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.