• mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
• attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.

MoE upcycling normally reuses only the dense model's feedforward layers. Carrying the attention parameters over too gives a better starting point for the same training budget.

• articleMultilingual Arbitrage Optimizing Data Pools To Accelerate Multilingual Progress 2024 08 28
• articleOne Tokenizer To Rule Them All Emergent Language Plasticity Via Multilingual Tokenizers 2025 05 30
• articleInclude Evaluating Multilingual Language Understanding With Regional Knowledge 2024 11 29
• articlePushing Mixture Of Experts To The Limit Extremely Parameter Efficient Moe For Instruction Tuning 2023 09 11

• articleSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient SparsityWilliam Fedus et al.
• articlePushing Mixture Of Experts To The Limit Extremely Parameter Efficient Moe For Instruction Tuning 2023 09 11Cohere editorial sitemap
• articleGlobal-batch load balance almost free lunch to improve your MoE LLM trainingQwen Team

Checking sign-in…

Loading comments…