• mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
• fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.

Mixture-of-experts routing over lightweight adapters means you can instruction-tune while updating a fraction of a percent of parameters and still match or beat full fine-tuning at similar scale.

• articleBam Just Like That Simple And Efficient Parameter Upcycling For Mixture Of Experts 2024 08 15
• articleMultilingual Arbitrage Optimizing Data Pools To Accelerate Multilingual Progress 2024 08 28
• articleOne Tokenizer To Rule Them All Emergent Language Plasticity Via Multilingual Tokenizers 2025 05 30
• articleInclude Evaluating Multilingual Language Understanding With Regional Knowledge 2024 11 29

• articleGlobal-batch load balance almost free lunch to improve your MoE LLM trainingQwen Team
• articleSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient SparsityWilliam Fedus et al.
• articleQwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated ParametersQwen Team

Checking sign-in…

Loading comments…