Kimi K3: The Manos, The Mythos, The Legendos
- Source
- newsletter.semianalysis.com
- Author
- Kimbo Chen
- Date
Context
A fixed-size working memory inside a model does not make the whole serving system's memory use constant. SemiAnalysis highlights this distinction in its Kimi K3 analysis. The model from Moonshot AI mixes layers that retain information per text unit with layers that continually update a fixed-size state.
The complication is reusing earlier work. As vLLM's implementation notes explain, that fixed-size state is overwritten as generation proceeds. Reusing an earlier prompt requires saving a snapshot before it changes. The serving engine therefore retains selected checkpoints, including prompt boundaries. More checkpoints consume memory but can avoid more recomputation on later requests, so serving efficiency depends on the retention policy as well as the model architecture.
- Kimi Delta Attention is best understood as online learning: linear attention compresses all past keys/values into one state matrix S, DeltaNet regularizes its unbounded growth via the delta rule, gating adds an LSTM-style forget gate, and KDA makes that gate a diagonal matrix for
- K3 interleaves KDA with full-attention MLA at a 3:1 ratio - the sweet spot Kimi Linear found - and KDA doubles as the position-aware operator, replacing RoPE entirely.
- FlashKDA achieves linear prefill and CONSTANT decode complexity in sequence length by unrolling the recurrence in chunks with a two-kernel design (chunk-tensor prep, then chunked recurrent compute).
- Keeping MLA instead of GQA is a deliberate decode-side bet: MLA absorption trades extra prefill compute for cheaper decode - sensible for reasoning workloads, costly for prefill-heavy agentic ones.
- Stability details that make it work: short convolutions on q/k/v for local dependencies, L2-normalized queries/keys, low-rank decay gates, and a per-head output forget gate.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
“We argue that one should not infer KV cache efficiency solely based on KV cache space complexity.”
“KV throughput represents the minimum bandwidth required to reliably serve a model with PD disaggregation, but it is also a good proxy for understanding KV cache efficiency.”
“As a result, all frontier open weight models use GQA-based attention mechanisms”
Checking sign-in…
Loading comments…