Modular LLMs at scale: how FlexOlmo is helping to pool national expertise without pooling sensitive data

FlexMoRE addresses a practical problem with combining independently trained model experts: storing many full-sized components becomes expensive. Building on Ai2's FlexOlmo architecture, independent Danish researchers kept some experts at full size and replaced most with compact, low-rank approximations. Ai2's account explains the approach.

The team found that reasoning tasks needed larger approximations than knowledge-focused tasks. Its best configuration reportedly exceeded a baseline of full-sized experts while using less than one-third as many parameters. This makes combining specialized contributions less memory-intensive, supporting the project's aim of bringing expertise together while leaving each contributor's original training data separate.