Linear attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → is not new. A 2020 paper replaced softmax with a kernel feature map so the n-by-n attention matrix is never built, cutting cost from quadratic to linear in sequence length. Those variants lost accuracy and never shipped in a frontier open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → model.
The 2025 reversal is the real evidence. MiniMax shipped M1 with lightning attention, then dropped it for the 230B M2, reporting linear attention held up on ordinary prompts but degraded on reasoning and multi-turn tasks, exactly what agents lean on.
Qwen3-Next's hybrid attention is what buys its native 262K-token context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →. The previous 235B model handled 32K natively and needed YaRN scaling to stretch to 131K.
DeepSeek V3.2's sparse attention is not strictly linear, but it is subquadratic, which puts it in the same efficiency family as MiniMax-M1, Qwen3-Next and Kimi Linear.
The author's own verdict after the tour: if you are shipping applications or fine-tuning, stay on proven decoder-style transformers. The alternatives are worth tracking, not worth betting a project on yet.
Terms in this piece · Glossary
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
If you track LLM architecture trends, this breaks down the concrete alternatives to standard autoregressive transformers—linear attention hybrids, Gated DeltaNet, text diffusion, and recursive small models—with code and clear diagrams groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → in real recent models like Qwen3-Next and Kimi Linear.
Key quotes
“The team stated that linear attention is tricky in production LLMs. It seemed to work fine with regular prompts, but it had poor accuracy in reasoning and multi-turn tasks, which are not only important for regular chat sessions but also agentic applications.”
“even if a diffusion model needs, say, 64 denoising steps to produce all tokens in parallel at each step, this is still computationally more efficient than performing 2,000 sequential generation steps to produce a 2,000-token response”
“So, I would maybe not call it a world model, but a world model-augmented LLM.”
“If I were to start a new LLM-based project today, autoregressive transformer-based LLMs would be my first choice.”
