Intel / article

Benchmarking LLM Inference at Scale with AIPerf

Source: developer.nvidia.com
Author: Elizabeth Goodman
Date: Published Sep 18, 2026

Context

A model-serving can underestimate a server if the program sending requests cannot generate enough load. NVIDIA's AIPerf addresses that measurement problem by separating request generation and result processing across multiple processes. It also lets engineers vary request timing and prompt lengths instead of testing only a steady stream of identical requests.

That matters because aggregate throughput and responsiveness describe different experiences. AIPerf reports how long the first response takes, gaps between generated text units, and completion time, including slow-end percentiles. Engineers can use those measurements to examine whether a deployment remains responsive under the traffic pattern they expect, rather than relying on one average speed figure.

Key takeaways · AI-distilled

AIPerf is a ground-up rewrite, not built on top of Perf Analyzer the way GenAI-Perf was; NVIDIA calls it a clean architectural break and provides a migration guide for porting existing GenAI-Perf workflows.
Its multiprocess design splits load-generating worker processes from separate record-processor services, coordinated over ZMQ, specifically so Python's GIL can't turn the benchmarking client itself into the bottleneck under high concurrency.
Beyond synthetic traffic, AIPerf replays real production traces via formats from Mooncake, Baseten, and WEKA AgentX, plus public datasets like ShareGPT, so benchmarks can mirror actual workload shape instead of a made-up one.
Load shape is controllable via constant, Poisson, and gamma arrival patterns with tunable burstiness, gradual concurrency ramping, and vLLM/SGLang range-ratio distributions for varying input/output sequence lengths.
Reported metrics include TTFT, inter- latency, request latency, and output token throughput broken down by percentile, plus GPU telemetry pulled in automatically when DCGM or pynvml is available.

Terms in this piece · Glossary

benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Read the source developer.nvidia.com