Intel / article

Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each

Source: developer.nvidia.com
Author: Elizabeth Goodman
Date: Published Sep 15, 2026

Context

A model's active parameter count describes how much of it participates in processing a piece of text, not how much memory its complete weights occupy. NVIDIA's dense-versus-MoE explanation uses Nemotron 3.5 Lightning as an example: it has 30 billion total parameters but activates about 3 billion per token.

A model selects a subset of its internal expert networks for each token. In the deployment described, the other experts still occupy GPU memory even while idle. This separates the storage budget from the computation used per token, so a small active count alone is insufficient to decide whether the model fits on available hardware.

Key takeaways · AI-distilled

MoE decouples memory from compute: all experts must sit in VRAM up front (scaling with total parameters), but only the routed experts run per token (scaling with active parameters), while dense models pay both costs together.
At roughly equal total parameters (~30B), Nemotron 3.5 Lightning's MoE+Mamba-2 hit 235-494 tok/s versus Gemma 4 31B's dense 37-222 tok/s at roughly half the price, but scored less than half as well on capability (24 vs 30 on Artificial Analysis's Intelligence Index).
MoE's throughput edge shrinks as concurrency rises: at batch size 1, decoding is memory-bound and MoE reads far fewer weight bytes per token, but as batch size grows, requests collectively touch most experts anyway, narrowing the advantage.
Full fine-tuning can unbalance an MoE router, making some experts dominant and others go unused; the fix is /PEFT-style tuning or freezing the router rather than a full fine-tune.
hits MoE and hybrid-attention models differently: in MoE, small precision loss in the router can flip routing decisions, so recipes keep router, , and output head at higher precision; hybrid-attention models need the recurrent projections protected instead.

Terms in this piece · Glossary

mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
LoRA — A cheap way to fine-tune a model by training a small add-on layer instead of changing all of the model's weights.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.

Read the source developer.nvidia.com