Intel / article

11 840 Tps On A Single Gpu

Source: cline.bot
Author: Cline evaluation and engineering blog
Date: Published Oct 30, 2026

Context

Cline, the AI coding , says it pushed NVIDIA's Nemotron 3.5 Super VL to 11,840 output per second on one GB300 in an NVIDIA DGX Station, serving 256 requests at once. That is nearly double the 6,058 tokens per second it got from the engine vLLM's base configuration on the same test of 512 input and 512 output tokens. The figure is aggregate throughput, counted across all 256 streams together, so it measures how much work one machine can serve, not how fast a single response arrives. Cline's walkthrough explains the settings it changed.

The gains came step by step. The model ships with a multi-token prediction head that drafts tokens ahead for the model to verify; using it with three draft tokens reached roughly 7,000 tokens per second. Asynchronous scheduling stopped the CPU's scheduling work from waiting on the GPU. The model mixes layers, which keep a cache, with Mamba layers, which carry a running state, and switching the Mamba kernel to FlashInfer reached 7,800, a result Cline says it cannot explain. Cutting the draft lookahead from three tokens to one then raised throughput to about 9,500: with 256 requests the GPU already had plenty of work, and each failed draft wasted model time. Storing the Mamba state at 16-bit instead of 32-bit precision brought the final step to 11,840.

Cline is clear that the recipe fits this traffic shape and the same tricks could lower throughput under other loads. It also says the 16-bit change may lower evaluation scores, and it reports no evaluation results. Its claim of a 66% saving from self-hosting compared with API pricing is described as napkin math that holds only when the GPU is fully used. The post shows what one machine can serve for one benchmark shape; it does not show answer quality or performance on real agent workloads.

Key takeaways · AI-distilled

Cline's baseline was a stock vLLM config at 6,058 aggregate tokens/sec (512 input and output tokens, 256 concurrent streams, 67 GB of memory) for NVIDIA's Nemotron 3.5 Super VL on one GB300 in a DGX Station.
Using the model's built-in MTP3 draft head lifted throughput to about 7,000 tokens/sec, async scheduling kept CPU scheduling from blocking GPU work, and switching the Mamba kernel to FlashInfer reached 7,800. Cline says it is unclear why the kernel change helped.
Cutting speculative lookahead from MTP3 to MTP1 raised throughput to 9.5K. At 256 concurrent requests the GPU already has plenty of work, so extra draft tokens that fail verification cost more time than they save.
The last step stored Mamba state in FP16 with stochastic rounding instead of FP32, reaching 11,840 tokens/sec, at a cost in precision that Cline notes could lower eval scores.
Cline concludes these settings depend on traffic shape and could hurt throughput under other load patterns. Its napkin math put a fully utilized self-hosted GPU at a 66% discount versus paying per API call.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.

Read the source cline.bot

More from Cline evaluation and engineering blog