AWS 'priming' builds efficient hybrid LLMs from Transformers

Priming is a way to build a hybrid language model by converting an existing Transformer instead of pretraining a new one. A hybrid keeps some attention layers, which store every past token in a memory (the KV cache) that grows with the length of the context, and replaces others with state space layers, which carry a fixed-size compressed summary of everything seen so far. That smaller memory is what makes hybrids faster on long inputs. Until now, the Amazon researchers say, trying a new hybrid design meant pretraining it from scratch on trillions of tokens, which kept most large-scale comparisons out of reach. In their paper, priming starts from a pretrained Transformer and needs less than 0.5% of the source model's pretraining token budget.

The method rests on a structural similarity: both attention and state space layers can be written as a matrix that decides how much each earlier token contributes to the current one. Where a state space layer can approximate an attention layer's matrix closely, the attention layer's learned query, key and value projections can be reused as the new layer's inputs, and only the parts specific to the state space layer have to be learned. Priming first picks which layers to convert, using a cheap test: layers that lose little when restricted to a short window of recent tokens are doing local work and are good candidates. It then copies the weights over, briefly trains each new layer to imitate the attention layer it replaced, and finally fine-tunes the whole model.

Because every variant starts from the same model, the team could compare three kinds of state space layer under identical conditions, with half of the sequence-mixing layers converted. Gated KalmaNet came out ahead of Gated DeltaNet, which beat Mamba-2, matching the order of how expressive their state updates are in theory. Every state space hybrid also beat a hybrid that used sliding-window attention instead, even when the window had two to four times as much memory. Gated KalmaNet can also run fewer solver iterations at serving time; the Amazon Science post says cutting from 30 to 10 iterations costs about 1% accuracy on AIME 2025 for 10% faster decoding.

The headline quality claim is narrower than "matches its source" may suggest. All reported results are on Qwen's Qwen3 at 8B and 32B; the authors say the method works across Transformer families such as Llama and Mistral and on mixture-of-experts models, but the paper reports no results for them. The 32B reasoning hybrid averages 63.0 across the paper's reasoning benchmarks, 3.8 points above off-the-shelf Qwen3-32B (59.2) but 1.1 points below the 64.1 of Qwen3-32B fine-tuned with the same recipe. The blog calls that within 1%; the paper's results section puts it at about 1.7% relative. In exchange, it needs roughly half the KV cache, which lets more requests run at once on the same GPU: with Gated KalmaNet running 10 solver iterations, decode throughput is 1.5 times the Transformer's at 16,000 tokens of context and about 2.1 times at 128,000, and on AIME 2025 it reached target accuracy 1.2 to 1.4 times as fast in wall-clock time. The models, training code, kernels and a vLLM serving plugin are released under Apache 2.0 on GitHub and Hugging Face.