FLOP/s comparisons understate newer chips for inference. Epoch estimates GB300 serves about 25x the requests of H100 per chip, which changes how you reason about future inference cost and capacity.
Key takeaways · AI-distilled
Epoch attributes the gap between FLOP/s and inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → capacity to memory and inter-chip communication. GB300 NVL72 scores about 1.9x B300 HGX with the same Blackwell Ultra GPUs because it links groups of 72 GPUs with high-bandwidth connections instead of eight.
The index counts how many requests a chip can serve at once across required speeds of 50 to 400 tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →/s per request, weighting each doubling equally, then averages the ratio to GB300 over four model architectures, including Kimi K3 and GLM-5.2.
Scores in GB300 equivalents: H200 HGX 0.12, B300 HGX 0.54, AMD MI355X about 0.55 and Vera Rubin VR200 2.18. For Kimi K3 alone the model predicts a roughly 46-fold GB300 advantage over H200, versus eightfold across the full basket.
Version 1 models agentic workloads: 262,144 tokens of cached context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → with 1,024 new tokens per prefill, about 1.51 new input tokens per output token, and speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition → assumed to accept three tokens per decoding cycle.
Epoch says the scores are modeled hardware potential and ignore software maturity. In its validation, GB300's measured Kimi K3 throughput tracked predictions assuming 50% of modeled peak, with an average absolute difference of about 12% across five points.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
