Benchmarking LLM Inference with NVIDIA AIPerf

AIPerf is NVIDIA's tool for load-testing a deployed language model, and its central design goal is that the tester itself should never be what limits the measurement. In NVIDIA's walkthrough, the authors argue that hand-rolled load scripts and most existing benchmarkers, including NVIDIA's own GenAI-Perf, run as a single process that becomes bound by Python's global interpreter lock under real concurrency. When the client hits that ceiling, the numbers describe the client's limit rather than the server's.

NVIDIA calls AIPerf the designated successor to GenAI-Perf and describes it as a ground-up rewrite that no longer runs on top of Perf Analyzer. Separate worker processes generate the load, separate services process the results, and the pieces coordinate over ZMQ, so the load generator can push a real server without becoming the bottleneck, according to NVIDIA. The tool supports more than 15 endpoint types, public datasets such as ShareGPT, replay of recorded traffic in formats from Mooncake, Baseten and WEKA, and constant, Poisson or gamma arrival patterns with tunable burstiness.

The post's own example shows why the shape of the load changes the answer. A first run on Qwen3-0.6B served through vLLM pins every request to 128 input and 128 output tokens, with flags that force the model to emit all 128 tokens, because a model that stops early produces throughput figures that are too low and do not repeat across runs. A second run sends an average of 10 requests per second at random, Poisson-distributed intervals, with prompt lengths varying around 512 tokens. Time to first token, the wait before a response starts arriving, spreads much more widely in that run because requests compete for the GPU and prefill and decode work overlap. That is why AIPerf reports every core metric at percentiles up to p99: the authors note that a server with a healthy average time to first token and an outlier p99 can look fine in aggregate and still fail in production.

The post explains how the tool works and what it measures, but it does not publish results comparing AIPerf against GenAI-Perf on the same server, so the size of the measurement error it removes is NVIDIA's argument rather than a demonstrated figure. Its clearest practical value is repeatability: a fixed random seed reproduces the same sequence of requests, and swapping in a different model or endpoint is a one-flag change, so a team can rerun the same realistic load after changing its setup and compare like with like.