Chinchilla showed that a fixed training budget could produce a stronger model by allocating more of it to training data. The 2022 study trained a 70-billion-parameter model using the same computational budget as the 280-billion-parameter Gopher, but four times as much data. Chinchilla performed better across the reported downstream evaluations. Experiments with more than 400 models supported increasing training data alongside model size.
This revisits the budget-allocation question in the 2020 scaling-law study. That earlier analysis favored much larger models trained on relatively modest datasets; Chinchilla demonstrated the value of a different balance.
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.