How NVIDIA tuned Nemotron 3 Ultra NIM for 2.5x more users
- Source
- developer.nvidia.com
- Author
- NVIDIAAI
- Date
Nvidia published the specific serving optimizations it applied to the Nemotron 3 Ultra NIM, covering caching strategy, memory layout, parallelism and decoding changes. Together they let four B200 GPUs handle 2.5 times as many concurrent users while holding steady at 50 tokens per second per user, meaning the gain is in throughput per dollar rather than a tradeoff against latency. The numbers matter because serving a large open model efficiently is mostly an engineering problem once the model itself is fixed, and Nvidia is showing its own working rather than a marketing claim. Teams running Nemotron or similar large open models can apply the same tuning categories to their own deployments, even without Nvidia's exact configuration, since the categories map to standard levers any inference stack exposes. It is a useful data point for anyone budgeting GPU capacity against expected concurrent load this quarter.
Context
NVIDIA says its ready-made serving package for Nemotron 3 Ultra handles up to 2.5 times as many simultaneous users on four B200 GPUs as an open-source serving setup without its optimizations, while each user still receives 50 tokens a second. In NVIDIA's engineering write-up, the optimized stack delivers 1,997 tokens a second of total output against 718 for the baseline, which works out to about 2.8 times, although NVIDIA labels the result 2.5x. The package is a NIM, a container that bundles the model with serving settings NVIDIA has tested for particular GPUs. The model has 550 billion parameters, of which 55 billion are active for each token, and mixes Mamba-2 and layers with some layers, according to its catalog page.
The comparison holds per-user speed fixed because serving more people at once normally slows each reply. NVIDIA set the target at 50 tokens a second per user, or 20 milliseconds between tokens, and measured how much total output fits under it on an agentic workload with long prompts and 76 percent reuse of previously computed . The write-up attributes the gain to settings that interact rather than add up: kernels autotuned for the model's expert and Mamba layers, splitting the model across all four GPUs, caching repeated prompt prefixes so shared context isn't recomputed, tuning how many requests the scheduler keeps in flight and how GPU memory is allocated, and multi-token prediction, a form of whose benefit NVIDIA says depends on how often drafted tokens are accepted and on spare memory.
The 2.5x figure is NVIDIA's own measurement on one hardware setup, one workload and one baseline, and it compares NIM against the same open-source serving stack without NIM's optimizations, not against other providers' optimized serving. NVIDIA itself calls the published curves a starting point rather than a promise, and recommends replaying representative traffic with its AIPerf tool to find the setting that meets an application's latency target.
- mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
Checking sign-in…
Loading comments…