Arcee opens Trinity Large weights, including an instruct-free base checkpoint
- Source
- x.com
- Date
Today, we’re releasing the first weights from Trinity Large, our first frontier-scale model in the Trinity MoE family.
We’re releasing three variants from the same training run: - Trinity-Large-Preview: lightly post-trained, chat-ready - Trinity-Large-Base: full 17T token pretraining checkpoint - Trinity-Large-TrueBase: 10T token checkpoint with no instruct data or LR anneals
Sparsity was a core design choice. By keeping the routing fraction low (4-of-256), Trinity Large achieves significantly faster inference than dense models in its weight class, while maintaining strong performance.
Context
Arcee released the first weights from Trinity Large, its first frontier-scale model in the Trinity family, trained on 2,048 NVIDIA B300 GPUs over a 17-trillion- run completed in 33 days. The model activates only 4 of 256 experts per token, a high degree of sparsity that Arcee says gives faster than a dense model of comparable total size.
Three checkpoints came from that same run: Trinity-Large-Preview, lightly post-trained and ready to use in a chat interface; Trinity-Large-Base, the full 17-trillion-token checkpoint; and Trinity-Large-TrueBase, an earlier 10-trillion-token snapshot with no instruction-tuning data and no final learning-rate anneal applied. Arcee frames TrueBase as a rarer artifact, since most models labeled 'base' already include some instruction data, and this one lets researchers study what large-scale pretraining alone produces.
To keep training stable at that level of sparsity, Arcee says it added extra dense, non-expert layers, used momentum-based load balancing so no single expert became overloaded with training examples, and applied a loss term called z-loss to control the scale of the model's output values.
- mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
Checking sign-in…
Loading comments…