• A widely used benchmark harness asked to sustain 200 queries/sec silently delivered only 38 and reported results as if it had hit the target — the load-testing tool itself was the bottleneck, not the server under test.
• Python's global interpreter lock caps a single-process load-generation harness around 170 QPS without warning, and one such thrashing client inflated its own measured latency by 58 seconds, which looked like a server problem but wasn't.
• A benchmark claiming 20% better throughput turned out to be running at temperature zero (deterministic, cheaper to compute) rather than the 0.7 setting real workloads use — a sampling-config mismatch disguised as a performance win.
• The same public dataset fed into two different benchmark harnesses produced different input token counts, because each harness sampled and truncated the text differently, undermining supposedly apples-to-apples comparisons.
• Google's fix, Inference Perf (a CNCF project), fans requests across separate worker processes that log when they actually fired versus when scheduled, so client telemetry can catch a failing harness instead of blaming the server — it held accurate accounting at 5,000 QPS.

• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
• benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
• agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.

Explains concrete, reproducible ways published LLM inference benchmarks mislead, including GIL-bound harnesses and mismatched temperature settings, directly relevant to anyone trusting third-party throughput numbers.

• videoOperating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta
• videoVertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave
• videoRouting LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI
• videoWhat's New in Inference Engineering — Philip Kiely, Baseten

• videoDeep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI ResearcherAI Engineer
• articleBenchmarking LLM Inference at Scale with AIPerfElizabeth Goodman
• videoKV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red HatAI Engineer

Checking sign-in…

Loading comments…