Optimally Allocating Compute Between Inference And Training

This 2024 essay by Ege Erdil, then a researcher at Epoch AI, argues that a lab minimizing its total compute bill should spend comparable amounts on training a model and on serving it (epoch.ai). It starts from a rough coincidence: Sam Altman had said OpenAI generated around 100 billion tokens a day, about 36 trillion a year, while modern models are trained on the order of 10 trillion tokens, and a training token costs around three times as much as an inference token because training needs backward passes as well as forward ones. On that naive arithmetic, annual inference and training costs come out at the same order of magnitude, and the essay asks why neither dominates.

The answer is the training-inference tradeoff. A lab can get better performance either by spending more on training or by spending more each time the model runs. Overtraining a smaller model on more data makes every token cheaper to serve; chain-of-thought prompting, web browsing and repeated sampling instead spend extra compute at run time. Citing Villalobos and Atkinson (2023), the essay takes as a rule of thumb that each technique can save about one order of magnitude of compute on one side in exchange for somewhat more than one order of magnitude on the other, and simplifies that to a one-for-one swap.

Given that swap, lopsided budgets are wasteful. In the essay's example, a lab spending 1 zettaFLOP on training and 100 on inference could train ten times harder and serve ten times more cheaply, keeping performance the same while spending 20 zettaFLOP instead of 101. More generally, if β orders of magnitude of training compute buy α orders of magnitude of inference savings, the cost-minimizing split puts a fraction α/(α+β) into training. The split is not very sensitive to the exact rate: for training and inference to differ by an order of magnitude, α would need to be below 0.1 or above 10, and Erdil says the empirical evidence does not support values that far from 1.

The essay names the main way this could fail: if every known inference-saving tradeoff is already used up and inference still dominates, there may be little left to trade. It also sets aside memory-bandwidth limits on inference, the falling cost of future compute, and its assumption that labs know demand in advance. Epoch's later breakdown of OpenAI's 2024 compute spending offers one real-world comparison: it estimated research and development compute well above inference spending, with most R&D compute going to research and experimental runs rather than final training runs, from investor figures it calls not completely reliable.