AIPerf is a ground-up rewrite, not built on top of Perf Analyzer the way GenAI-Perf was; NVIDIA calls it a clean architectural break and provides a migration guide for porting existing GenAI-Perf workflows.
Its multiprocess design splits load-generating worker processes from separate record-processor services, coordinated over ZMQ, specifically so Python's GIL can't turn the benchmarking client itself into the bottleneck under high concurrency.
Beyond synthetic traffic, AIPerf replays real production traces via formats from Mooncake, Baseten, and WEKA AgentX, plus public datasets like ShareGPT, so benchmarks can mirror actual workload shape instead of a made-up one.
Load shape is controllable via constant, Poisson, and gamma arrival patterns with tunable burstiness, gradual concurrency ramping, and vLLM/SGLang range-ratio distributions for varying input/output sequence lengths.
Reported metrics include TTFT, inter-tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → latency, request latency, and output token throughput broken down by percentile, plus GPU telemetry pulled in automatically when DCGM or pynvml is available.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
Gives engineers a way to benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition →inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → under production-like bursty traffic without the benchmarking client becoming the limiting factor, with percentile latency and GPU telemetry built in.
