Agentic Inference Capacity Growth
- Source
- epoch.ai
- Author
- Epoch AI publications
- Date

Context
Over the year to mid-2026, the AI chips sold to date gained capacity to serve AI agents much faster than they gained raw computing power. Epoch AI estimates that between the second quarter of 2025 and the second quarter of 2026, the -serving capacity of all AI chips sold to date grew 4.6 times, while their peak computing power, measured in 8-bit operations per second, grew 2.6 times. The gap comes from what limits , the work of running a trained model to answer requests. Epoch says memory and communication between chips can matter more than arithmetic speed, and in long agent sessions the cached of each conversation has to stay in memory and be read again as the model works, so arithmetic speed alone understates how much a newer chip can serve.
To measure that, Epoch uses its Inference Performance Index, a modeled score of how many simultaneous requests a chip can serve on a fixed set of agentic workloads with long, cached contexts. The unit is the GB300 equivalent: NVIDIA's GB300 scores 1, so a chip scoring 2 handles about twice as many users or agents. Epoch multiplies each chip family's score by its estimated cumulative sales and adds them up, keeping the same workloads for every quarter.
Most of the gain came from one hardware transition. NVIDIA's Blackwell and Blackwell Ultra chips account for 74% of the capacity added over the year, and a GB300 system delivers roughly 8 times the inference capacity of an H200, the previous generation, per GPU on these workloads. Google's TPUs, AMD GPUs and Amazon's Trainium chips contribute smaller shares. Across the whole fleet, inference capacity per unit of peak compute rose about 77%.
Epoch expects the pace to slow. Chips sold in the second quarter of 2026 added only 3.3 times the capacity added a year earlier, less than the 4.6x growth of the total, and Epoch's model puts the per-GPU step from GB300 to NVIDIA's next Vera Rubin VR200 at 2.2x rather than 8x. Faster shipments could make up some of the difference. The estimate's 90% range of 4.2 to 5.1 times covers uncertainty in shipment counts and hardware mix only; the per-chip scores are modeled rather than measured, software efficiency is not modeled, and the split between rack-scale and server Blackwell systems is Epoch's judgment.
- Epoch AI estimates the agentic inference capacity of AI chips sold to date grew 4.6x from Q2 2025 to Q2 2026 (90% interval 4.2-5.1x), against 2.6x for peak 8-bit compute, measured on fixed workloads with long, cached contexts.
- The Hopper-to-Blackwell transition drives most of the gain: Epoch models a GB300 at about 8x the per-GPU inference capacity of an H200, and Blackwell families supplied 74% of the added capacity.
- Epoch expects growth to slow: chips sold in Q2 2026 added 3.3x the capacity of Q2 2025 sales, below the 4.6x total growth, and it models only a 2.2x per-GPU gain from GB300 to Vera Rubin VR200.
- Inference capacity per unit of peak compute rose about 77% in a year on these workloads, which Epoch attributes to hardware-model co-design: larger scale-up domains, lower-precision math and model architectures that use the hardware better.
- Epoch's limits: the 60/40 split between GB rack systems and HGX servers is a judgment call, software efficiency is not modeled, and its uncertainty intervals cover shipments and hardware mix but not the per-chip inference scores.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Checking sign-in…
Loading comments…






