Arcee's open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → 400B mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition → activates only 13B parameters per token and reaches 256K context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →. The report documents load-balancing and data-loading methods that avoided loss spikes, which is useful for teams training or self-hosting large sparse models.
Key takeaways · AI-distilled
Trinity Large has 400B total parameters but activates only 13B per token: four of 256 routed experts plus a shared expert, with a 3:1 mix of sliding-window and global attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → layers to cut long-sequence cost.
Arcee fixed early router instability and expert collapse with SMEBU, momentum-damped expert bias updates, and a data loader (RSDB) that cut batch heterogeneity 4.23x and step-to-step loss variance 2.4x in the final phase.
Extending only the global attention layers took Trinity Large to 256K tokens; it scored 0.994 on multi-key needle-in-a-haystack at 256K and 0.976 at an untrained 512K, but fell to 0.42 at 1M.
Arcee reports Trinity Large Base is competitive with GLM-4.5 Base with roughly 2.5x fewer active parameters, and calls the instruct Preview preliminary because its post-training was intentionally light.
Terms in this piece · Glossary
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
