Optimally Allocating Compute Between Inference And Training
Source
epoch.ai
Author
Epoch AI publications
Date
Why it matters
Explains why training and inference spend should land at similar magnitudes, which helps in reasoning about how labs price tokens and allocate compute.
Key takeaways · AI-distilled
Ege Erdil notes Sam Altman claimed OpenAI generates ~100B tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → a day (~36T a year); with models trained on ~10T tokens at ~3x the per-token cost, a naive analysis suggests OpenAI's annual inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → and training costs are of the same order.
Erdil argues rough parity is expected because of the training-inference compute tradeoff: a combination of techniques such as overtraining, chain-of-thoughtHaving a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.Full definition →, repeated sampling and more MCTS playouts can let a smaller model match a larger one.
Estimates Erdil cites from Villalobos and Atkinson (2023) vary by technique: repeated sampling with cheap verification has a max tradeoff span of about 3 orders of magnitude in training and 6 in inference, versus about 1.6 each for MCTS.
Erdil's parity conclusion assumes labs can flexibly shift compute between training and inference while holding performance constant; he says it would most plausibly fail if the known tradeoffs 'run out of steam'.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
chain-of-thought — Having a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.
