one matrix replaced the KV cache.
(the technique is 100% open source)
Kimi just dropped K3, an open model at frontier scale.
it leans on a new mechanism called delta attention that does not keep a growing KV cache. that is how it holds a million tokens of context without the memory blowing up.
before we can understand delta attention, we need to understand attention itself.
it is a lookup. every token stores a key, which works like an address, and a value, which is the content at that address. to build its output, a token sends out a query, matches it against every key in the sequence, and pulls back a blend of the values whose keys matched. that is the picture at the top of the diagram.
standard attention keeps every one of those key and value pairs as a list, one entry per token. that list is the KV cache, and it grows with the sequence, so each new token has to scan the whole thing to build its output. double the context and you double both the storage and the scanning, which is where the quadratic cost and the ballooning memory come from.
delta attention keeps the lookup but throws away the list. the entire past collapses into one fixed-size matrix that still behaves…
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → mechanisms that eliminate KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition → growth change the cost model for long-context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition →.
Read the full post on X ↗
