Intro to Mixture of Experts | Aritra Roy Gosthipaty | HF Podcast #2
- Source
- youtube.com
- Author
- Hugging Face
- Date
Context
Aritra Roy Gosthipaty, a developer advocate on Hugging Face's Transformers team, explains (MoE) in a Hugging Face podcast. In a dense model, every parameter takes part in generating each , a chunk of text. In an MoE, the feed-forward layers are split into smaller expert networks and only a subset runs for each token. Gosthipaty's illustrative figure is a trillion-parameter model activating about 20 percent per token, which Gosthipaty says saves roughly 80 percent of compute and helps speed and efficiency.
Dense models are not going away, Gosthipaty adds: small dense models, often from big MoEs, suit phones and other edge devices, while foundation models increasingly use MoE. The cost named is that MoEs are hard to train. The 20 percent figure is an example, not a specification for any particular model.
- mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
- Token — A piece of text, such as a word, part of a word or a punctuation mark, that a language model represents as numbers and handles as a unit.
- Inference — Using a trained model to produce an output from a supplied input.
- distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
Checking sign-in…
Loading comments…






