Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI
Source
AI Engineer
Author
AI Engineer
Date
Key takeaways · AI-distilled
OpenAI's old inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → router used a proportional feedback loop: shift traffic off a hot engine, it cools down, the controller reads that as spare capacity and routes traffic back — the resulting oscillation wrecked the KV-cache locality the routing was meant to protect.
The old weighting system had no clean explanation for its own decisions: asking why one engine got a higher weight had no good answer, and tuning one property would move another unpredictably.
The replacement splits into a control plane with a global view of every cluster and a data plane that answers 'which engine serves this request' from a cached snapshot of weights — signals now feed an optimizer rather than a live feedback loop.
The new optimizer minimizes expected end-to-end latency, not just distance: in OpenAI's example, a nearby engine already serving 120 requests/sec against a 100/sec capacity loses out to one two regions away sitting at 40 of 80, because the queueing wait outweighs the network hop.
Production safeguards include outlier penalties, retry budgets that tighten automatically as utilization climbs (to head off retry storms), and load shedding as the last resort when the above aren't enough.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
A concrete architecture for fixing KV-cache-destroying oscillation in LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → load balancing, useful for anyone operating multi-engine inference fleets at scale.
videoOperating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta
videoVertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave
videoAre LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google
videoWhat's New in Inference Engineering — Philip Kiely, Baseten
