Shoutout to the cracked team at @vllm_project that implemented recent agentic workload optimizations. (1/5)🧵
Agents reread their whole history every turn. Reuse the model's cache of it and you skip that work. Saving that cache off-GPU was wasteful: agents each saved their own copy, and every turn re-saved everything from scratch. Now, one save per shared prefix, and each turn writes only what's new. (2/5)
@vllm_project Another optimization lets split-across-machines serving hand over a hybrid model's complete state, so the decode half doesn't start with amnesia. All of this came from replaying real agent traffic. (3/5)
@vllm_project With these and other optimizations, the team at vLLM brought Kimi inference to new heights, increasing high-concurrency throughput by more than 6x in 2 weeks. (4/5)
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Why it matters
AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → loops reread their full history every turn, so KV-cache reuse is the dominant serving cost. Deduplicating cache offload by shared prefix and passing hybrid-model state across disaggregated workers cut that overhead enough to raise high-concurrency throughput several times over.
Read the full thread on X ↗
