DeepSeek V4.1 Flash gets up to 1.5x faster on GB300 via software
- Source
- x.com
- Date
DeepSeek V4.1 Flash speeds on GB300 NVL72 have increased by up to 1.5x tok/s per agent over the past month on AA-AgentPerf, our benchmark of datacenter systems running realistic agentic coding workloads This improvement comes from a newer vLLM version, new vLLM kernels for DeepSeek V4.1 on GB300, FlashInfer autotuning, fuller CUDA graph coverage, tuned speculative decoding, and running prefill and decode on the same GPUs in NVIDIA’s updated Dynamo + vLLM setup ➤ Faster speeds: 848 tok/s for a single agent, up from 596 (1.42x) ➤ Higher throughput: ~1.6x the throughput per GPU at ~80 tok/s per agent
DeepSeek V4.1 Flash on GB300 NVL72 rose from 596 to 848 tok/s for a single through software changes alone. Vera Rubin delivers 5.0x GB300 throughput per GPU at 260 tok/s per agent, which bears on agent serving cost.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
Checking sign-in…
Loading comments…