FlexMoRE shrinks modular training to commodity hardware
- Source
- x.com
- Date
The Danish Foundation Models (DFM) project is adapting our modular FlexOlmo architecture into a lighter-weight system that runs on commodity hardware—putting collaborative model building within reach of smaller research groups & organizations. 🧵
FlexOlmo lets teams train modules separately then combine them in a shared model without pooling the data underneath. DFM wants Danish institutions like hospitals & universities to each contribute modules trained on data they can't share.
In FlexOlmo, each module is the size of a full model, so the combined system grows fast as more get added. FlexMoRE replaces most modules with compact representations. Its best config matches or beats FlexOlmo using less than one-third the parameters.
"FlexMoRE significantly reduces FlexOlmo's memory demands while preserving performance across almost all categories, allowing a broader audience to benefit from modular models," says Jacob Nielsen, who helped develop FlexMoRE at Ordbogen A/S and SDU's OdenseNLP lab.
Context
Combining independently trained model components becomes expensive when every component is full-sized. FlexMoRE, developed independently by Danish researchers using Ai2's FlexOlmo, replaces most full-sized components with compact approximations of their differences from a shared base model. This reduces the memory needed to hold the combined model. Project account.
The research compressed existing experts after training. Its best configuration, which gave each expert its own adapter size, used 10.75 billion parameters against 33.27 billion for the full-sized baseline, and scored a modestly higher average (47.18 against 45.46) across six groups. The best size varied by task, and the sizes were chosen using one of the benchmarks that is also reported, so the margin should be read cautiously. Training compact experts from scratch was not tested. Methods and limitations.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Checking sign-in…
Loading comments…