• Meta's inference traffic already exceeds its largest microservices and is its fastest-growing workload, forcing orchestration (routing, KV cache management, prefill/decode disaggregation) to consolidate the way cloud infrastructure did around schedulers and autoscalers.
• A single routing decision cascades: it changes cache hit rate, which changes batch composition, which changes GPU utilization, which then changes what the autoscaler does next — coupling that classic microservice architectures don't have to manage.
• Preempting a GPU mid-request after 200 tokens have already streamed can't just be retried, so Meta argues reliability has to live in a control plane that's the only layer with visibility into the whole request workflow.
• A production inference scheduler needs visibility into seven axes at once — GPU generation, memory headroom, KV cache state, warm weights, tenant priority, latency budget, and workflow context — failing step three of five wastes what the first two already spent.
• Meta frames every inference optimization as one of four moves: avoid the work (caching), share it (batching), move it (routing to a cheaper model or region), or delay it (admission control) — and the metric to optimize is cost per successful task, not per token.

• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
• KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
• AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
• context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Lays out a seven-axis scheduling model (GPU generation, KV cache state, warm weights, tenant priority) for reliable large-scale inference, useful for teams operating agent workloads in production.

• videoVertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave
• videoAre LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google
• videoRouting LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI
• videoWhat's New in Inference Engineering — Philip Kiely, Baseten

• videoVertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeaveAI Engineer
• videoRouting LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAIAI Engineer
• videoWhat's New in Inference Engineering — Philip Kiely, BasetenAI Engineer

Checking sign-in…

Loading comments…