Nexus Specialization Meets Adaptability For Efficiently Training Mixture Of Experts 2024 08 29

Nexus explores how to add a new specialist to a language-model system without repeating large-scale training of the entire mixture. The publisher abstract from Cohere describes combining separately trained models with an adaptive routing mechanism that determines how experts are used.

The proposed mechanism represents experts using information about their domains. That allows another separately trained expert to join the system, followed by limited fine-tuning. The authors report gains when extending the mixture this way. The useful research direction is incremental specialization: adapting a collection of models to a new domain without rebuilding the whole collection through large-scale mixture training.