[Paper Club] Upcycling Large Language Models into Mixture of Experts

A trained language model can become the starting point for a larger mixture-of-experts model. In this research presentation, an NVIDIA researcher explains upcycling: copying parts of an existing network into expert branches, then training a router to select which branches process each piece of text.

Because only some branches activate for each input, the model can add capacity without activating every added parameter. The researchers report that their upcycling experiments outperformed continued training of the original dense models. Initialization is central to their method: the conversion must preserve useful behavior closely enough for further training to build on it.