Explains how latency and cost trade off in serving LLMs and why faster serving is more valuable for long agentic reasoning loops, so you can anticipate pricing and speed tiers.
Key takeaways · AI-distilled
Epoch's model breaks each transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition → forward pass into arithmetic time, HBM read-write time, network send-receive time, and fixed latency such as kernel launches and collectives.
Fixed latency matters even for tiny transfers: NCCL's LL128 all-reduce has a 30-microsecond base latency on a DGX H100 even for a one-byte tensor.
Since some components can overlap, the model assumes overlap and grid-searches setups for best speed at fixed cost. For dense Transformers it finds tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → generation speed scales roughly with 1/sqrt(parameters) and the cube root of GPU memory bandwidth, a rule far from exact.
Ege Erdil argues 10 tokens per second at human reading speed is an obsolete bar: long reasoning and AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → loops make faster serving more valuable, while inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → revenue at OpenAI and Anthropic grows 3x or more a year.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
