Intel / article

Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention

Source: magazine.sebastianraschka.com
Author: Sebastian Raschka, PhD
Date: Published May 16, 2026

Context

A language model can shrink the memory it needs for long contexts by reusing information across layers. Google's Gemma 4 E2B and E4B models apply this approach to the stored key-value representations used to process earlier text.

In Sebastian Raschka's architecture walkthrough, later layers reuse keys and values from an earlier layer of the same attention type while still computing their own queries. Each layer can therefore form its own attention pattern without maintaining a separate copy of all that cached information. Building on ordinary KV caching, this reduces the memory burden of keeping long contexts available.

Key takeaways · AI-distilled

Gemma 4 E2B has 35 layers but only the first 15 compute their own key/value projections; the last 20 reuse KV from the nearest earlier layer of the same attention type. That roughly halves the cache, about 2.7 GB saved at a 128K .
Layers that share KV still compute their own queries, so each one forms its own attention pattern; only the memory-heavy keys and values are recycled. The cost is reduced capacity, which the cross-layer attention paper found minimal at small scale.
Per-layer embeddings explain why Gemma 4 E2B is quoted as both 2.3B and 5.1B parameters. The extra weight sits in embedding tables looked up once per token and injected as a small per-layer slice, so the stack stays as cheap as a 2B model.
The reason architecture work swung toward and memory traffic: reasoning models and agents keep far more tokens around for far longer, so long-context memory rather than raw compute became the binding constraint.

Terms in this piece · Glossary

attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.

Key quotes

“For the smallest E2B model, this results in a 2.7 GB saving (at bfloat16 precision) in long 128K contexts, as shown below.”

“The DeepSeek V4 paper reports that, at a 1M-token context length, DeepSeek V4-Pro uses only 27% of the single-token inference FLOPs and 10% of the KV cache size compared with DeepSeek V3.2, which uses MLA and DeepSeek Sparse Attention (DSA).”

“While it was possible to implement a basic transformer block in perhaps 50-100 lines of PyTorch code, these tweaks (esp. around the attention variants) probably 10x the code complexity.”

“The moral of the story, I guess, is to keep learning, one architecture at a time :).”

Read the source magazine.sebastianraschka.com