Can we change an AI model’s architecture without retraining it from scratch?
Source
amazon.science
Date
Why it matters
Priming starts hybrid sparse-attention and state-space models from pretrained transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition → weights, avoiding trillions of training tokens. That makes memory-efficient long-session architectures far cheaper to test and may lower inference costs for agents.
Key takeaways · AI-distilled
Priming has three stages: pick layers that can be compressed into a fixed-size state, initialize those SSM layers from the attention weights and distill each one, then fine-tune the whole hybrid. The authors say this uses under 0.5% of the source pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → tokens.
To decide which attention layers to convert, the team swaps each one for sliding-window attention and measures the drop; layers that tolerate it mostly do local work and suit a state space layer.
In controlled comparisons Gated KalmaNet beat Gated DeltaNet, which beat Mamba-2, and every SSM hybrid outperformed sliding-window attention hybrids on long context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → even when the window got two to four times the memory.
Gated KalmaNet's iteration count can be changed at serving time without retraining: the authors report 10 iterations instead of 30 costs about 1% accuracy on AIME 2025 for 10% faster decoding.
Amazon reports the 32B hybrid's throughput edge grows from 1.50x at 16K tokens to 2.06x at 128K, and a training-free chunkingSplitting documents into passages small enough to embed and retrieve individually — the step that quietly determines whether retrieval works at all.Full definition → method that merges SSM states extends context from 128K to 256K tokens.
Terms in this piece · Glossary
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
chunking — Splitting documents into passages small enough to embed and retrieve individually — the step that quietly determines whether retrieval works at all.
