• mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.

Lets you build a mixture-of-experts model out of dense checkpoints you already trained, and extend it with new experts later instead of retraining the whole system when a new domain arrives.

• articleBam Just Like That Simple And Efficient Parameter Upcycling For Mixture Of Experts 2024 08 15
• articleMultilingual Arbitrage Optimizing Data Pools To Accelerate Multilingual Progress 2024 08 28
• articleOne Tokenizer To Rule Them All Emergent Language Plasticity Via Multilingual Tokenizers 2025 05 30
• articleInclude Evaluating Multilingual Language Understanding With Regional Knowledge 2024 11 29

• articleGlobal-batch load balance almost free lunch to improve your MoE LLM trainingQwen Team
• articlePushing Mixture Of Experts To The Limit Extremely Parameter Efficient Moe For Instruction Tuning 2023 09 11Cohere editorial sitemap
• articleSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient SparsityWilliam Fedus et al.

Checking sign-in…

Loading comments…