Kimi Delta attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → is best understood as online learning: linear attention compresses all past keys/values into one state matrix S, DeltaNet regularizes its unbounded growth via the delta rule, gating adds an LSTM-style forget gate, and KDA makes that gate a diagonal matrix for
K3 interleaves KDA with full-attention MLA at a 3:1 ratio - the sweet spot Kimi Linear found - and KDA doubles as the position-aware operator, replacing RoPE entirely.
FlashKDA achieves linear prefill and CONSTANT decode complexity in sequence length by unrolling the recurrence in chunks with a two-kernel design (chunk-tensor prep, then chunked recurrent compute).
Keeping MLA instead of GQA is a deliberate decode-side bet: MLA absorption trades extra prefill compute for cheaper decode - sensible for reasoning workloads, costly for prefill-heavy agentic ones.
Stability details that make it work: short convolutions on q/k/v for local dependencies, L2-normalized queries/keys, low-rank decay gates, and a per-head output forget gate.
Terms in this piece · Glossary
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
If you're building or serving hybrid linear/full-attention models, this explains the FLOP and memory-traffic tradeoffs of Kimi Delta Attention and why linear attention complicates prefix caching in engines like vLLM — concrete architecture-level knowledge for anyone optimizing inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → infra.
Key quotes
“We argue that one should not infer KV cache efficiency solely based on KV cache space complexity.”
“KV throughput represents the minimum bandwidth required to reliably serve a model with PD disaggregation, but it is also a good proxy for understanding KV cache efficiency.”
“As a result, all frontier open weight models use GQA-based attention mechanisms”
