The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI
Source
youtube.com
Author
AI Engineer
Date
Context
An AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → can repeatedly send much of the same conversation history as it works through a task. Model-serving company FriendliAI argues that this makes reuse of previous computation central to agent performance. Its presentation's published account describes storing the model's calculations for a shared beginning of the input and reusing them on later calls.
Routing then becomes part of the saving: a later request goes to a server that already holds the reusable calculations. Spreading requests evenly without considering that stored work can force repeated computation. The design treats consecutive calls as parts of one continuing task, so the relevant outcome is how quickly the whole task finishes.
Key takeaways · AI-distilled
Chun's team originated continuous batching, now standard across the serving industry, and their follow-on work inspired one of the most widely used open-source LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → serving frameworks.
The demo task was the same coding agent building a tower defense game, run once on a closed frontier model and once on an open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → model; both finished at a usable level.
Beyond the three caching/routing techniques, FriendliAI adds a fourth pillar: agent-aware scheduling that treats an incoming call as part of a longer running program rather than an isolated request.
One customer's split test found the agent-optimized stack seven times faster with a lower error rate than its prior setup.
Chun's argument for why request latency is the wrong metric: an agent's real unit of work is a plan-act-observe loop that can run minutes to hours, with sub-agents fanning out in parallel and every observation appended to a context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → that only grows.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
videoWhy AI Makes You 100x Faster but Not Your Company - Mitchell Troyanovsky, Basis
videoBuilding the AI Agent Is Easy. Proving It's Safe Isn't - Jonathan Osborne, NatWest
videoDon't Approve the AI Model. Approve the App - Doneyli De Jesus, Langfuse
videoWithout Caching, Your AI Agent Costs 10x More - George Cameron, Artificial Analysis
