• Multi-head latent attention squeezes key/value tensors into a smaller space before they enter the KV cache, then expands them back at inference. One extra matrix multiply buys the memory saving, and DeepSeek-V2's ablations had it beating plain multi-head attention on quality.
• DeepSeek V3 carries 671B parameters but runs only 37B per token: each mixture-of-experts layer holds 256 expert feed-forward blocks, of which a router picks 8, plus one shared expert that fires for every token.
• The always-on shared expert exists so every specialist doesn't relearn the same common patterns. It absorbs the generic work and leaves the routed experts room to specialize, an idea that traces back to the 2022 DeepSpeedMoE paper.
• Seven years of architecture progress is largely substitutions on the same GPT skeleton: absolute position embeddings became rotary (RoPE), multi-head attention became grouped-query attention, GELU became SwiGLU. The block diagram barely moved.

• open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
• mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
• KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
• embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.

If you're building on or evaluating open-weight LLMs, this gives you side-by-side architectural breakdowns (MLA vs GQA, MoE routing variants, normalization placement) with code, so you can reason about the real tradeoffs behind DeepSeek-V3, Kimi K2, Gemma, and OLMo 2 instead of trusting benchmark headlines.

• articleA Dream of Spring for Open-Weight LLMs: 10 Architectures from Jan-Feb 2026
• articleBeyond Standard LLMs
• articleFrom GPT-2 to gpt-oss: Analyzing the Architectural Advances
• articleUnderstanding the 4 Main Approaches to LLM Evaluation (From Scratch)

• articleA Dream of Spring for Open-Weight LLMs: 10 Architectures from Jan-Feb 2026Sebastian Raschka, PhD
• articleRecent Developments in LLM Architectures: KV Sharing, mHC, and Compressed AttentionSebastian Raschka, PhD
• articleBeyond Standard LLMsSebastian Raschka, PhD

Checking sign-in…

Loading comments…