EMO: Pretraining mixture of experts for emergent modularity
Source
allenai.org
Date
Terms in this piece · Glossary
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
Why it matters
If a task only needs one capability, EMO lets you load an eighth of the experts at near full quality, offering a route to serving large mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition → models without provisioning memory for parameters the workload never touches.
articlePushing Mixture Of Experts To The Limit Extremely Parameter Efficient Moe For Instruction Tuning 2023 09 11Cohere editorial sitemap