Vibeleaderboard
← All Intel
Intel / video

Intro to Mixture of Experts | Aritra Roy Gosthipaty | HF Podcast #2

Source
youtube.com
Author
Hugging Face
Date

Context

Aritra Roy Gosthipaty, a developer advocate on Hugging Face's Transformers team, explains (MoE) in a Hugging Face podcast. In a dense model, every parameter takes part in generating each , a chunk of text. In an MoE, the feed-forward layers are split into smaller expert networks and only a subset runs for each token. Gosthipaty's illustrative figure is a trillion-parameter model activating about 20 percent per token, which Gosthipaty says saves roughly 80 percent of compute and helps speed and efficiency.

Dense models are not going away, Gosthipaty adds: small dense models, often from big MoEs, suit phones and other edge devices, while foundation models increasingly use MoE. The cost named is that MoEs are hard to train. The 20 percent figure is an example, not a specification for any particular model.

Terms in this piece · Glossary
  • mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
  • Token — A piece of text, such as a word, part of a word or a punctuation mark, that a language model represents as numbers and handles as a unit.
  • Inference — Using a trained model to produce an output from a supplied input.
  • distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
Read the source www.youtube.com
More from Hugging Face
Recommended reads
Comments

Checking sign-in…

Loading comments…