Inference Economics Of Language Models

Language models are not served much faster than they are, even for customers willing to pay more per token, mainly because of fixed communication delays between GPUs, according to a model of inference economics by Ege Erdil, published by Epoch AI in 2025 with an accompanying arXiv paper (epoch.ai). The model splits each forward pass of a transformer into four costs: arithmetic on the GPU cores, moving data from high-bandwidth memory into those cores, sending data between GPUs over the network, and latency, the fixed overhead of steps such as kernel launches and collective operations that does not shrink with message size. Erdil gives the example that one NVIDIA all-reduce protocol on a DGX H100 has a base latency of 30 microseconds even for a one-byte tensor. Searching over ways to split a model across GPUs then traces the best achievable speed at each price.

The key finding is about which of those costs binds. Network bandwidth gets more attention, but Erdil argues it only dominates for transfers above about 10 MB, which would need per-GPU batch sizes of 500 or more, far larger than anyone would use when serving for speed. In fast inference the network cost is therefore mostly latency. Without that constraint, spreading a forward pass across more GPUs could keep quadrupling speed for each doubling of cost per token, a point the post credits to Steinhardt (2022). Because latency does bind, that path runs out. As rough order-of-magnitude rules, the model finds a dense transformer's token speed scales with the inverse square root of its parameter count and the cube root of the GPUs' memory bandwidth.

The same accounting explains why speculative decoding, where a cheap draft proposes several tokens and the large model checks them in a single pass (background), pays off so well. It does nothing for arithmetic or network bandwidth, but in fast decoding those are not the limits; memory bandwidth and network latency are, and both get spread across several tokens per pass. Erdil reports the effect can be as large as doubling generation speed at fixed cost. The model also implies pipeline parallelism, splitting a model's layers across GPUs in sequence, is mainly worth using when a model would not otherwise fit in GPU memory, which on an H100 means speeds below roughly 42 tokens per second.