NVIDIA Vera Rubin NVL72 Leads Agentic Inference Performance on MLPerf Endpoints v1.0
- Source
- developer.nvidia.com
- Author
- Zhihan Jiang
- Date

Context
NVIDIA's claim that its new Vera Rubin NVL72 rack serves AI agents with over ten times the throughput per kilowatt of its current GB300 NVL72 rests on one pair of measurements taken at a fixed speed for each user. MLPerf Endpoints v1.0, a benchmark from the industry group MLCommons, sends coding and enterprise sessions to a system through an API and checks the generated tool calls against reference answers so that speed is not bought with wrong output. A serving system faces a tradeoff: handling more users at once raises the total it produces per second, but each user's response arrives more slowly. NVIDIA's comparison fixes a floor of 50 tokens per second for each user and asks how much total output each rack produces per kilowatt of power while staying above it.
The footnote in NVIDIA's post gives the two runs behind the headline, both on Kimi K3, a model from Moonshot AI. The Vera Rubin system, with 32 accelerators, handled 1,024 simultaneous queries and produced 41,414.2 tokens per second on 100.89 kW, with users averaging 50.4 tokens per second. The GB300 system, with 48 accelerators, handled 48 simultaneous queries and produced 2,210.4 tokens per second on 60 kW, with users averaging 56.2. That works out to roughly 410 against 37 tokens per second per kilowatt, about 11 times. Most of the gap comes from Vera Rubin holding the speed floor while serving more than 20 times as many queries at once as the GB300 run NVIDIA compared it with.
The comparison holds per-user generation speed fixed, but not the wait before an answer starts. At those same two operating points, the 90th-percentile time to first token, a wait before any output appears that 90 percent of requests stayed under, was 4,673.9 milliseconds on Vera Rubin and 2,305.1 on GB300, about twice as long. NVIDIA's table across three models reports 10x, 9x and 5x the throughput per kilowatt on Kimi K3, DeepSeek-V4.1-Flash and Qwen3.6-35B-A3B, and 3x to 5x faster per-user speed when both racks serve the same number of queries, so the advantage is large but its size varies by model and by which measure is held equal.
These are NVIDIA's own submissions, described by NVIDIA. The GB300 results are verified by MLCommons; the Vera Rubin results are provisional, pending MLCommons verification, and may change after peer review. An independent benchmark points the same direction with smaller ratios: in October, Artificial Analysis reported that Vera Rubin served up to about 2.8 times GB300's throughput per megawatt on Kimi K3 and about 5.4 times on DeepSeek V4.1 Flash in its own agentic test (earlier Intel on those results), under different conditions, so the figures do not line up directly. The supported conclusion is that Vera Rubin can carry far more agent traffic within a given power budget at this speed floor, while the over-10x figure applies to one model, one threshold and one pair of runs that still await verification.
- On Kimi K3, NVIDIA's provisional Vera Rubin NVL72 submission served 41,414 tokens/s at 100.89 kW versus GB300 NVL72's 2,210 tokens/s at 60 kW, both at about 50 tokens/s per user. That is the basis of its over-10x throughput-per-kilowatt claim, pending MLCommons verification.
- The gain varies by model: throughput per provisioned kilowatt rose 10x on Kimi K3, 9x on DeepSeek-V4.1-Flash and 5x on Qwen3.6-35B-A3B, while end-to-end interactivity at matched concurrency rose 3x, 5x and 3x.
- MLPerf Endpoints v1.0 measures serving through API endpoints on multi-turn coding and enterprise workflows, checks generated tool calls against reference values during runs, and uses separate runs to test whether faster serving preserves model quality.
- NVIDIA credits much of the gain to software that exploits agent reuse: prefix caching and Dynamo routing to avoid recomputing shared history, disaggregated prefill and decode, and model-specific work such as DeepSeek's MegaMoE kernel and Qwen's multi-token prediction.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Checking sign-in…
Loading comments…


