Intel / article

Beyond Standard LLMs

Source: magazine.sebastianraschka.com
Author: Sebastian Raschka, PhD
Date: Published Nov 4, 2025

Context

Some language models reduce the memory cost of long conversations by combining two ways of handling earlier text. In Sebastian Raschka's explanation of alternative architectures, Gated DeltaNet layers maintain a fixed-size state that is updated as text arrives. Conventional layers retain access to individual earlier positions.

Qwen's Qwen3-Next combines these mechanisms, placing three Gated DeltaNet layers before each full-attention layer, as its model card documents. The fixed-size state limits memory growth in the recurrent layers while the remaining attention layers preserve direct access to past text. The design therefore reduces some long- costs without making the entire model's memory use constant.

Key takeaways · AI-distilled

Linear attention is not new. A 2020 paper replaced softmax with a kernel feature map so the n-by-n attention matrix is never built, cutting cost from quadratic to linear in sequence length. Those variants lost accuracy and never shipped in a frontier model.
The 2025 reversal is the real evidence. MiniMax shipped M1 with lightning attention, then dropped it for the 230B M2, reporting linear attention held up on ordinary prompts but degraded on reasoning and multi-turn tasks, exactly what agents lean on.
Qwen3-Next's hybrid attention is what buys its native 262K-token context. The previous 235B model handled 32K natively and needed YaRN scaling to stretch to 131K.
DeepSeek V3.2's sparse attention is not strictly linear, but it is subquadratic, which puts it in the same efficiency family as MiniMax-M1, Qwen3-Next and Kimi Linear.
The author's own verdict after the tour: if you are shipping applications or , stay on proven decoder-style transformers. The alternatives are worth tracking, not worth betting a project on yet.

Terms in this piece · Glossary

attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.

Key quotes

“The team stated that linear attention is tricky in production LLMs. It seemed to work fine with regular prompts, but it had poor accuracy in reasoning and multi-turn tasks, which are not only important for regular chat sessions but also agentic applications.”

“even if a diffusion model needs, say, 64 denoising steps to produce all tokens in parallel at each step, this is still computationally more efficient than performing 2,000 sequential generation steps to produce a 2,000-token response”

“So, I would maybe not call it a world model, but a world model-augmented LLM.”

“If I were to start a new LLM-based project today, autoregressive transformer-based LLMs would be my first choice.”

Read the source magazine.sebastianraschka.com