The Danish Foundation Models (DFM) project is adapting our modular FlexOlmo architecture into a lighter-weight system that runs on commodity hardware—putting collaborative model building within reach of smaller research groups & organizations. 🧵
FlexOlmo lets teams train modules separately then combine them in a shared model without pooling the data underneath.
DFM wants Danish institutions like hospitals & universities to each contribute modules trained on data they can't share.
In FlexOlmo, each module is the size of a full model, so the combined system grows fast as more get added.
FlexMoRE replaces most modules with compact representations. Its best config matches or beats FlexOlmo using less than one-third the parameters.
"FlexMoRE significantly reduces FlexOlmo's memory demands while preserving performance across almost all categories, allowing a broader audience to benefit from modular models," says Jacob Nielsen, who helped develop FlexMoRE at Ordbogen A/S and SDU's OdenseNLP lab.
Modular training lets institutions contribute modules without pooling private data, but FlexOlmo modules are each model-sized. FlexMoRE compresses them to under a third of the parameters while holding performance, bringing collaborative training onto commodity hardware.
Read the full thread on X ↗
