The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
When a model generates token 5,001, it needs to relate that token to all 5,000 before it. The KV cache stores each earlier token's internal representation (its "keys and values") so they're computed once, not once per new token. Without it, generation would slow to a crawl.
The catch is size: the cache grows with context length and eats GPU memory, which limits how many users one GPU can serve. That's why so much inference engineering — and so many papers in this index — is really KV-cache engineering: compressing it, sharing it across requests, pricing cached tokens cheaper.