Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Source
William Fedus et al.
Author
William Fedus et al.
Published
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
The paper that made mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition → practical, by routing each tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → to exactly one expert instead of blending many. Every frontier model advertising a huge parameter count with a small active count runs this idea — which is why parameter count and inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → cost stopped being the same conversation.
