Thoughts About Scaling Law
Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions.
The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed.
Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today…
Kaplan et al.'s exponent specifically told teams to grow parameters roughly 2.7x faster than data; Hoffmann et al.'s 2022 Chinchilla study retested this across 400 models before landing on the ~20-tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →-per-parameter rule.
Concretely, that over-training shows up as Llama-2-7B trained at roughly 290 tokens per parameter and Gemma-2-9B at roughly 889 — far beyond Chinchilla's optimum for training-only cost.
In mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition → models, total parameters roughly govern how much knowledge the model can hold while activated parameters and effective depth govern how far it can reason, so a dense model's tokens-per-parameter ratio doesn't transfer directly to MoE.
Cited research finds the optimal tokens-per-parameter ratio is task-dependent: memorization favors more parameters, reasoning favors more data, and at a fixed ratio, raising total parameters can hurt reasoning while activating more experts helps it.
GLM-5.3 is presented as a controlled test of this theory: same base model, architecture, and parameter counts as GLM-5.2, with the only change being a month of additional long-horizon RL post-training.
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
Why it matters
Explains why parameter count alone is meaningless without data volume, compute budget, and deployment context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →, and why models today are deliberately over-trained relative to Chinchilla because inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition →, not training, dominates lifetime cost.
Read the full post on X ↗
