SemiAnalysis: Agentic Workloads Now Over 70% of Inference Traffic
- Source
- x.com
- Date
SemiAnalysis reports that agentic workloads now account for more than 70 percent of all LLM inference traffic, up from a market that was dominated by single-turn chat requests. Agentic sessions look structurally different: they run multi-turn, accumulate long context over the course of a task, spawn sub-agents in bursts, and repeatedly reuse the same prefix of a conversation across many calls.
That shift explains why inference providers have spent the past year optimizing specifically for KV-cache reuse and prefix caching rather than raw throughput on isolated requests. A provider whose infrastructure was tuned for stateless chat is now serving a majority workload of long, stateful, bursty sessions, and the economics of serving that traffic depend on how much of a session's cache can be kept warm and reused rather than recomputed. For teams building agents, this is also a cost lever worth understanding: structuring prompts and tool calls to preserve a stable, reusable prefix is aligned with where the majority of inference spend now sits.
AGENTIC TRAFFIC NOW MAKES UP MORE THAN 70% OF ALL INFERENCE TRAFFIC 🚀 Agentic workloads are characterized by four elements: 🟠 Multi-turn: a session includes tens or hundreds of turns, leading to high potential KV-cache reuse. 🟠 Long context: system prompts, tool definitions, and the large number of turns make context accumulate quickly. 🟠 High prefix reuse: since the conversation progresses linearly, where output from turn n-1 is concatenated to turn n (typically), most context can be served from KV cache rather than recomputed (this depends on the amount of storage available to store KV tensors). As n grows, the ratio of cached input relative to uncached input typically tends towards 1. 🟠 Sub-agent bursts: a session launches multiple short-lived sub-agents with fresh context, which create bursty KV-cache patterns.
Context
SemiAnalysis, a semiconductor and AI infrastructure research firm, reports that agentic workloads, sessions where a model calls tools and takes many turns to complete a task, now account for more than 70% of all inference traffic. In its post, SemiAnalysis explains why this shift matters for cost: each new turn in an session typically appends to the same growing conversation, so most of the tokens a model processes were already computed in an earlier turn. Serving infrastructure can reuse that stored computation, called the , instead of recomputing it, and SemiAnalysis says that reused fraction tends toward nearly all of the input as a session grows longer. That is the reasoning behind SemiAnalysis's own earlier account of recent serving-engine changes built specifically to save and reuse that cache across an agent session's turns.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- subagent — A separate agent spawned by another to do one scoped piece of work in its own context, returning only the result.
Checking sign-in…
Loading comments…