A case study on how Claude achieves 92% cache hit-rate Every time an AI agent takes a step, it sends the entire conversation history back to the LLM. That includes the system instructions, the tool definitions, and the project context it already processed three turns ago. All of it gets re-read, re-processed, and re-billed on every single turn. For long-running agentic workflows, this redundant computation is often the most expensive line item in your entire AI infrastructure. A system prompt with 20,000 tokens running over 50 turns means 1 million tokens of redundant computation billed at full price, producing zero new value. And that cost compounds across every user and every session. The fix is prompt caching. But to use it well, you need to understand what’s actually happening under the hood. Static vs. Dynamic context Before you can optimize a prompt, you need to understand what changes and what doesn’t. Every agent request has two fundamentally different parts: The static prefix that stays identical across turns: system instructions, tool definitions, project context, and behavioral guidelines. The dynamic suffix that grows with every turn: user messages, assistant responses, tool outputs, and terminal observations. This split is what makes prompt caching possible. The infrastructure stores the mathematical state of the static prefix so that subsequent requests sharing that exact prefix can skip the computation entirely and read from memory. Once you internalize this, every architectural decision in this article becomes obvious. How does the KV Cache work? To understand why caching is so effective, you need to know what the transformer actually does when it processes your prompt. Every LLM inference request has two phases: The prefill phase handles the entire input prompt. It runs dense matrix multiplications across all tokens in context to build the model’s internal representation. This is compute-bound and expensive. The decode phase generates tokens one at a time. Each new token gets added to the sequence, and the model predicts the next one. This phase is memory-bound because it mostly reads the historical state rather than doing heavy computation. During the prefill phase, the transformer computes three vectors for each token: a Query, a Key, and a Value. The attention mechanism uses these to determine how each token relates to every other token. The Key and Value vectors for any given token depend only on the tokens before it, and once computed, they never change. Without caching, these Key and Value tensors get thrown away after every request, and the next request recomputes them from scratch. For a 20,000-token prefix, that’s 20,000 tokens worth of attention computation that didn’t need to happen again. The KV cache fixes this by persisting those tensors on the inference servers, indexed by a cryptographic hash of the token sequence. When a new request comes in with the same prefix, the hash matches, the tensors are loaded from memory, and the prefill computation for those tokens is skipped entirely. This drops computational complexity from O(n²) per generated token to O(n). And for a 20,000-token prefix repeated across 50 turns, that's an enormous reduction. The Economics The pricing structure is what makes this architectural decision so consequential. Cache reads cost 0.1x the base input price, which is a 90% discount on every cached token. Cache writes cost 1.25x, a 25% premium to store the KV tensors. Extended one-hour caching costs 2.0x. Here’s what this looks like across Anthropic’s Claude models: This math only works if the cache hit rate stays high. The best production example of what that looks like is Claude Code. A 30-minute coding session with Claude Code Claude Code is built entirely around one objective: keep the cache hot. Here’s what a real 30-minute coding session looks like from a billing perspective. Minute 0: Claude Code loads its system prompt, tool definitions, and the project’s CLAUDE.md file. This payload exceeds 20,000 tokens, and since every token is new, this is the most expensive moment of the entire session. But you only pay this cost once. Minutes 1 to 5: You start giving instructions, and Claude Code dispatches its Explore Subagent to navigate the codebase, open files, and run grep commands. All of this gets appended to the dynamic suffix. But the 20,000-token static prefix is now reading from cache at $0.30/MTok instead of $3.00/MTok. Minutes 6 to 15: The Plan Subagent receives a summarized brief rather than the raw results, because passing raw output would bloat the dynamic suffix unnecessarily. It produces an implementation plan, you approve it, and Claude Code starts making changes. Every turn reads the static prefix from cache, the hit rate climbs past 90%, and each access resets the TTL to keep the cache warm. Minutes 16 to 25: You request changes, which means more tool calls, more terminal output, and more context accumulating in the dynamic