A lot of work goes into serving a model efficiently.
For the Nemotron 3 Ultra NIM, our engineers tuned caching, memory, parallelism, decoding and more. On four B200 GPUs, those optimizations supported up to 2.5x more concurrent users while maintaining 50 TPS/user.
Read the engineering deep dive →
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
Concrete numbers on what it takes to serve a large open model efficiently: the specific caching, parallelism and decoding tuning that doubled concurrent user capacity without hurting per-user speed.
