Trading Off Compute In Training And Inference

In a July 2023 report, Epoch AI researchers Pablo Villalobos and David Atkinson asked how far compute spent running a model can substitute for compute spent training it. Training compute is the usual yardstick for capability, and some policy proposals rely on it. But some techniques raise performance by spending more on each answer, such as sampling many attempts, tree search or chain-of-thought reasoning, while others, such as pruning a model or training a smaller one on more data, cut the cost of each answer by spending more up front.

The authors fitted scaling data for four techniques: changing the balance of model size and training data, Monte Carlo tree search in the board game Hex, pruning, and repeated sampling on code and math problems. Their rule of thumb is that each technique saves about one order of magnitude (a factor of 10) of one kind of compute in exchange for somewhat more than one order of magnitude of the other. Overtraining, for example, can match Chinchilla while spending 2 orders of magnitude more on training to save 1 on inference; resampling can match AlphaCode while spending 1.5 orders more on inference to save 1 on training. The exception is tasks where answers can be checked cheaply and failures cost little, such as passing automated tests or producing formal proofs. There, they estimate, 6 more orders of magnitude of inference could replace 3 orders of training, a regime they had observed only in coding and formal mathematics. Combining techniques, they expected savings of up to about 2-3 orders of magnitude, because methods that work through the same mechanism interfere with each other.

The governance point follows from economics. Inference dominates the cost of a widely deployed model, so the authors expected companies to push public models toward the cheap-to-run end, while more capable, inference-heavy versions could exist for internal use or a few customers. In some cases they estimated extra inference could match a model trained with 2 orders of magnitude more compute, roughly the gap between GPT-3 and GPT-4 before counting algorithmic progress. Evaluations and safety research could use this to anticipate capabilities, and companies could use augmented models to speed up their own research.

These are 2023 estimates built around GPT-4 as the frontier reference, from limited data: the authors say their model for unlimited sampling should not be taken very seriously, and they did not quantify chain of thought. They describe the shape of the tradeoff rather than current figures. Epoch's analysis of how to split compute between inference and training takes up the related allocation question.