vLLM on Rubin NVL72 reports 6.5x perf per TCO over GB300

Inference cost on next-generation hardware may drop sharply: a reported 6.5x perf per TCO gain for Rubin NVL72 over GB300 on a MoE model with vLLM. It affects future token pricing and availability, but the submission details were not read.

• mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
• token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

• postGDPx: a frontier agentic benchmark where the top score is 12%
• postTPUv7 posts first agentic inference benchmark, with perf-per-dollar claims
• postClaude Code idle compaction targets cache-expiry costs
• postGoogle TPUv7 posts first agentic inference benchmark result

• articleVera Rubin NVL72 vs GB200 NVL72? Inference TCO & Architecture AnalysisAlec Ibarra
• blogNVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 DebutZhihan Jiang
• articleVera Rubin NVL72 Agentic Inference: 67x better Performance per DollarBryan Shan

Checking sign-in…

Loading comments…