• mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
• fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.

If you're training or fine-tuning MoE models, this shows how to apply load balancing at the global-batch level to keep expert utilization even — reducing routing collapse with minimal added cost compared to standard per-batch auxiliary losses.

• articleChinese CLIP: Contrastive Vision-Language Pretraining in Chinese
• articleOFASys: Enabling Multitask Learning with One Line of Code!
• articleQwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated Parameters
• articleGeneralizing an LLM from 8k to 1M Context using Qwen-Agent

• articlePushing Mixture Of Experts To The Limit Extremely Parameter Efficient Moe For Instruction Tuning 2023 09 11Cohere
• articleQwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated ParametersQwen Team
• articleSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient SparsityWilliam Fedus et al.

Checking sign-in…

Loading comments…