Intel / video

Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI

Source: youtube.com
Author: AI Engineer
Date: Published Sep 19, 2026

From the Daily Brief

OpenAI engineers Qianru Lao and Lu Zhang described replacing their production LLM inference router, previously built on a feedback-loop design, after it proved prone to unexplainable oscillation between backend engines. Oscillating traffic is especially costly for LLM serving because it destroys KV-cache locality: every time a request bounces to a different engine, the cached context has to be rebuilt from scratch, which erases most of the latency and cost benefit of caching in the first place. Their fix splits routing into a control plane and a data plane, treating request routing explicitly as a constrained latency-optimization problem instead of letting it emerge from reactive load signals. For teams operating multi-engine inference fleets, especially anyone serving agentic workloads with long, cache-heavy contexts, the talk lays out a concrete architectural pattern for keeping routing decisions stable without sacrificing the load-balancing benefits routing exists to provide.

Read the 2026-09-21 Brief →

Context

OpenAI's routing talk describes why moving requests away from a busy server can create instability. Once traffic leaves, the server looks underused, attracting requests back and causing repeated swings that disrupt reuse of cached computation. The publisher's account says this motivated a replacement based on explicit latency optimization.

The replacement considers both network travel and waiting time, subject to server-capacity limits. A more distant server can therefore be preferable when the nearby one has a longer queue. This supplies a concrete example of the interacting routing and caching decisions discussed in the distributed-inference talk.

Key takeaways · AI-distilled

OpenAI's old router used a proportional feedback loop: shift traffic off a hot engine, it cools down, the controller reads that as spare capacity and routes traffic back — the resulting oscillation wrecked the KV-cache locality the routing was meant to protect.
The old weighting system had no clean explanation for its own decisions: asking why one engine got a higher weight had no good answer, and tuning one property would move another unpredictably.
The replacement splits into a control plane with a global view of every cluster and a data plane that answers 'which engine serves this request' from a cached snapshot of weights — signals now feed an optimizer rather than a live feedback loop.
The new optimizer minimizes expected end-to-end latency, not just distance: in OpenAI's example, a nearby engine already serving 120 requests/sec against a 100/sec capacity loses out to one two regions away sitting at 40 of 80, because the queueing wait outweighs the network hop.
Production safeguards include outlier penalties, retry budgets that tighten automatically as utilization climbs (to head off retry storms), and load shedding as the last resort when the above aren't enough.

Terms in this piece · Glossary

inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.

Read the source www.youtube.com