Vibeleaderboard
← All Intel
Intel / article

Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality

Source
arxiv.org
Author
Zeyuan Allen-Zhu
Date
Why it matters

Architecture gaps vanish at academic scale but show up at 1-2T tokens, and local token mixing (Canon layers) may explain much of what newer linear models add. That affects which architectures are worth trusting.

Key takeaways · AI-distilled
  • In academic-scale (1.3B parameters, 100B ), carefully tuned architectures performed almost alike. The differences became measurable only at 1-2T tokens, across 64 from-scratch runs of up to 8B parameters at roughly $100k per 8B run.
  • Canon layers, which add local token mixing, saved about 30% of pretraining tokens for the same MMLU accuracy at 8B. With them, plain gated linear matched Gated DeltaNet and beat Mamba2, so the author argues their extra machinery may mostly mix neighboring tokens.
  • Benchmark parity hid basic failures: linear models that matched Transformers on standard benchmarks could not reliably retrieve information from a 500-token , and no model reliably beat random guessing on a short two-hop question, even at 8B/1T.
  • Small synthetic playgrounds under 100M parameters predicted these architectural differences. The author's practical advice is to put in-context reasoning examples into pretraining data early, rather than saving them for mid-training or post-training.
Terms in this piece · Glossary
  • pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Recommended reads
Comments

Checking sign-in…

Loading comments…