Vibeleaderboard
← All Intel
Intel / article

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Source
arxiv.org
Author
William Fedus et al.
Date

Context

Switch Transformers increase a model's total capacity while activating only selected groups of parameters for an input. These groups are called experts, and a routing mechanism chooses which ones participate. The 2021 paper simplifies that routing and addresses communication costs and unstable training. Its T5-Base and T5-Large variants achieved up to sevenfold speed improvements with the same computational resources in the reported experiments. This explains how a model can contain an enormous number of parameters without using all of them on every input: total stored capacity and active computation are different quantities.

Terms in this piece · Glossary
  • pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Recommended reads
Comments

Checking sign-in…

Loading comments…