Intel / article

A Dream of Spring for Open-Weight LLMs: 10 Architectures from Jan-Feb 2026

Source: magazine.sebastianraschka.com
Author: Sebastian Raschka, PhD
Date: Published Feb 25, 2026

Context

A model's active parameter count describes how much of it participates in each generation step, not the size of the whole model. Sebastian Raschka's architecture survey illustrates this with Qwen's Qwen3-Coder-Next. Its official model card specifies 80 billion total parameters but only 3 billion activated per step.

The model routes work through selected expert components, allowing a large collection of learned parameters to participate selectively. The smaller active count therefore does not describe the memory needed to store the model. Raschka reports that his local version took about 48.2 GB of storage and used about 51 GB of RAM. For someone considering local deployment, that separates the computation headline from the hardware requirement.

Key takeaways · AI-distilled

Sliding-window lets each token look only at the last ~4,096 tokens instead of a 256k , cutting per-layer cost from O(n squared) to roughly O(n times window). Arcee's Trinity runs 3 local layers per global one.
Trinity multiplies the attention output by a learned elementwise gate before the final projection. That damps attention sinks — positions that soak up attention for no reason — and improved both long-sequence behavior and training stability.
Trinity initializes the second RMSNorm gain in each block at about 1/sqrt(L), where L is the layer count. Residual updates start tiny and grow as training finds the right scale — a stability trick hiding inside a Gemma 3-looking norm layout.
Kimi K2.5's ablation: for a fixed budget of image tokens, mixing a smaller number in early during beats dumping more in later. When the model first sees vision matters more than how much of it there is.
Multi-token prediction bolts small extra heads onto each position to predict tokens t+1 through t+4 and sums the losses. The extra signal speeds training while inference still emits one token at a time; Step 3.5 Flash hits 100 tokens/sec at 128k context.

Terms in this piece · Glossary

mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.

Key quotes

“Step 3.5 Flash is a 196B parameter model that is more than 3x smaller than the recent DeepSeek V3.2 model (671B) while being slightly ahead in modeling performance benchmarks.”

“The GLM-5 release on February 12th was a big deal, because at the time of its release it appeared to be on par with the major flagship LLM offerings, including GPT-5.2 extra-high, Gemini Pro 3, and Claude 4.6 Opus.”

“So, the more likely issue here is that the SWE-Bench Verified benchmark has saturated, and it may no longer be a meaningful benchmark to report from now on (in favor of other benchmarks like SWE-Bench Pro, for example).”

“Modeling performance is likely not attributed to the architecture design itself but rather the dataset quality and training recipes (a good topic for a separate article).”

Read the source magazine.sebastianraschka.com