one matrix replaced the KV cache.
- Source
- x.com
- Date
one matrix replaced the KV cache. (the technique is 100% open source) Kimi just dropped K3, an open model at frontier scale. it leans on a new mechanism called delta attention that does not keep a growing KV cache. that is how it holds a million tokens of context without the memory blowing up. before we can understand delta attention, we need to understand attention itself. it is a lookup. every token stores a key, which works like an address, and a value, which is the content at that address. to build its output, a token sends out a query, matches it against every key in the sequence, and pulls back a blend of the values whose keys matched. that is the picture at the top of the diagram. standard attention keeps every one of those key and value pairs as a list, one entry per token. that list is the KV cache, and it grows with the sequence, so each new token has to scan the whole thing to build its output. double the context and you double both the storage and the scanning, which is where the quadratic cost and the ballooning memory come from. delta attention keeps the lookup but throws away the list. the entire past collapses into one fixed-size matrix that still behaves…
Context
Kimi K3 reduces part of the memory burden of long conversations by changing how some layers retain earlier text. Conventional stores a key and value for each text unit, creating a growing cache. Kimi Delta Attention instead updates a compact internal state as the sequence advances. Moonshot AI identifies this mechanism as part of K3's architecture.
The fixed-size memory in the post's diagram describes that recurrent mechanism, not the entire model. The serving implementation account confirms that K3 also retains full-attention layers and their growing caches. The model therefore combines compressed memory with conventional attention; the diagram does not establish constant memory use for K3 as a whole.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Checking sign-in…
Loading comments…