← Glossary
Glossary · models
mixture-of-experts
Also: MoE, mixture of experts
A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
A dense model uses all its parameters on every token. A mixture-of-experts model contains many "experts" and a router that picks perhaps 2 of 64 for each token — so a model can have a trillion parameters total but only run a fraction of them per step.
That's the trick behind many frontier and open-weight models: capability tends to scale with total parameters while cost scales with active ones. The trade: the whole thing must still sit in memory, and routing adds its own engineering (load balancing, training stability).
In the index
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- [Paper Club] Upcycling Large Language Models into Mixture of Experts
- Pushing Mixture Of Experts To The Limit Extremely Parameter Efficient Moe For Instruction Tuning 2023 09 11
- Global-batch load balance almost free lunch to improve your MoE LLM training
- Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each