FlexMoRE makes modular model training fit commodity hardware

Ai2's FlexOlmo lets separate groups each train a specialist part of one model, called an expert, on their own data, and then combine the experts without sharing that data. A router decides which experts handle each piece of text. The catch is size: every expert is full size, so a model that combines many of them needs a lot of memory. FlexMoRE, built by researchers at the University of Southern Denmark and the Danish company Ordbogen as part of the Danish Foundation Models project, keeps FlexOlmo's structure but turns most experts into low-rank adapters. An adapter stores a compressed record of how an expert differs from the shared public model, and its rank sets how much of that difference it keeps.

The FlexMoRE paper made these adapters without further training. It took six existing FlexOlmo experts, computed how each differs from the public model, and kept only the largest components of that difference. Its best mixture used 10.75 billion parameters against 33.27 billion for a baseline of full-size FlexOlmo experts, and averaged 47.18 against 45.46 across six groups of benchmarks. The right adapter size varied by expert and task: reasoning-heavy benchmarks needed larger adapters than knowledge tests. The sizes in that best mixture were chosen using one of the six benchmark groups, so part of the margin may come from that choice, and training small experts from scratch was not tested.

Ai2's thread presents the work as bringing collaborative model building within reach of smaller research groups on commodity hardware. The memory saving is measured in the paper, which reports per-expert reductions from about 31% for the math expert to about 96% to 98% for code, creative writing and Reddit, and more than 99% for the academic and news experts. Neither the paper nor Ai2's account of the project, also covered in a longer Intel item, runs the system on named consumer hardware. The Danish team developed FlexMoRE independently, using FlexOlmo's open release.