Chat-style inference benchmarks miss the long-prefix, cache-heavy traffic of coding agents. Agents per megawatt gives a more relevant measure of serving capacity and cost.
Key takeaways · AI-distilled
AA-AgentPerf replays real coding-AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → trajectories and finds the maximum number of concurrent agents a system can sustain while meeting per-agent output speed and time-to-first-tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → targets, reported as Agents per Megawatt.
Its trajectories run up to 200 turns, with inputs from about 5K to 131K tokens per request (mean about 27K, mostly tool output and history) and outputs that are mostly short tool calls and edits, spanning 12+ languages.
The full test set stays private; participants get a representative tuning subset so they can validate configurations without optimizing for the held-out benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →.
In the first results, NVIDIA Blackwell systems led with a large generational jump over Hopper, and rack-scale deployments showed advantages from scale and disaggregated inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → in both compute and power efficiency.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
