Bam Just Like That Simple And Efficient Parameter Upcycling For Mixture Of Experts 2024 08 15

BAM is a method for building a mixture-of-experts model, which routes each input to specialized sub-networks, out of several dense models that were already trained separately. Cohere's research page for the paper says training such models from scratch at large scale is prohibitively expensive, and that earlier "upcycling" reused only each dense model's feed-forward layers while merging the other parameters. BAM also reuses the attention parameters of each specialist, initializing them into a soft mixture-of-attention layer.

The abstract reports that, on starting models from 590 million to 2 billion parameters, BAM surpasses baselines in both perplexity (how well the model predicts text) and downstream task performance, within the same computational and data constraints. The evidence is the abstract only, at those small scales; full-paper methods and larger-scale results are not covered here. The practical point is a way to reuse existing training work when assembling a mixture-of-experts model, not a demonstrated large-scale deployment.