A Technical Tour of the DeepSeek Models from V3 to V3.2
- Source
- magazine.sebastianraschka.com
- Author
- Sebastian Raschka, PhD
- Date
Context
DeepSeek's V3.2 reduces the expensive work applied to long text by selecting which earlier positions deserve closer processing. Its technical report describes a lightweight indexer that scores earlier tokens, followed by a selector that retrieves the highest-scoring entries for the main attention calculation.
Sebastian Raschka's technical tour contrasts this with a sliding window, which selects nearby text by position. Learned selection can reach relevant material farther back in the . The savings come from restricting the main attention calculation to that subset; the indexer still performs its own scoring work.
- Multi-head latent attention squeezes keys and values into a smaller latent space before they enter the , then projects them back up when used. You pay an extra matrix multiply to buy a much smaller memory footprint.
- The reinforcement-learning ladder, plainly: classic needs a reward model and a critic. GRPO drops the critic. RLVR with GRPO drops the reward model too, scoring answers with a compiler or calculator instead of learned human preference.
- Hybrid versus dedicated reasoning models keeps flip-flopping. Qwen3 started hybrid then split into separate instruct and reasoning models because each was easier to train and better at its job; DeepSeek went the other way, R1 to hybrid V3.1 and V3.2.
- DeepSeek shipped V3.2-Exp months before V3.2 for a practical reason: its sparse attention variant needs custom inference code, so the experimental release existed to get serving infrastructure ready for the real model.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
- RLHF — Reinforcement learning from human feedback — training a model to prefer answers humans rate as better, which turns a raw text predictor into a usable assistant.
“DeepSeek-V3.2 uses exactly the same architecture as DeepSeek-V3.2-Exp”
“For reasoning and agent tasks, we employ rule-based outcome reward, length penalty, and language consistency reward. For general tasks, we employ a generative reward model where each prompt has its own rubrics for evaluation.”
“All experiments used a single model, our final proof generator, which performs both proof generation and verification.”
Checking sign-in…
Loading comments…