The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI
Source
AI Engineer
Author
AI Engineer
Date
Key takeaways · AI-distilled
Chun's team originated continuous batching, now standard across the serving industry, and their follow-on work inspired one of the most widely used open-source LLM serving frameworks.
The demo task was the same coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → building a tower defense game, run once on a closed frontier model and once on an open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → model; both finished at a usable level.
Beyond the three caching/routing techniques, FriendliAI adds a fourth pillar: agent-aware scheduling that treats an incoming call as part of a longer running program rather than an isolated request.
One customer's split test found the agent-optimized stack seven times faster with a lower error rate than its prior setup.
Chun's argument for why request latency is the wrong metric: an agent's real unit of work is a plan-act-observe loop that can run minutes to hours, with sub-agents fanning out in parallel and every observation appended to a context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → that only grows.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
FriendliAI's Byung-Gon Chun argues agent inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → should be measured by task completion, not request latency, and shows prefix caching, hierarchical KV storage, and cache-aware routing making an open-weight coding agent 5.5x cheaper than a closed frontier model.
videoWhat's New in Inference Engineering — Philip Kiely, Baseten
videoLarge clusters for small models — Daniel Svonava, Superlinked
videoWeight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli
videoTwo Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer
