Multi-teacher on-policy distillationTraining a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.Full definition → works like this: train several domain-specialist teachers, then have one general student generate its own rollouts and match, tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → by token, the output distribution of whichever teacher owns that domain.
It exists because one big RL run mixing math, code, and agentic tasks eventually trades those capabilities against each other. Specialists trained separately are cheap, well-understood, and do not fight.
The scale-up is fast: MiMo Flash v2 used about six teachers, DeepSeek V4 and Nemotron 3 Ultra push past ten, and Nemotron runs the distillation twice, re-distilling from refreshed teachers.
Llama 3 used its reward model only as a filter, never for online RL: sample K completions per prompt, reject the bad ones, SFT, DPO, repeat six rounds with the best models seeding the next.
Modern post-training is bounded by the org chart as much as by compute. OLMo 3's recipe stayed simple partly because coordinating more stages was already at the limit of what the organization could wrangle.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
It gives ML practitioners a clear, comparative map of how post-training recipes differ across DeepSeek, Llama, Tülü, OLMo, Nemotron, Kimi, and GLM, and explains the shift toward multi-teacher on-policy distillation — useful if you're designing or evaluating your own fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → pipeline.
Key quotes
“The shape of a post-training recipe has changed more in the last year than in the prior three.”
“post-training OLMO-3 to make this reasoning model was a major accomplishment for many individuals to do this. But also, the complexity of what we were doing was pushing against the limits of AI2\”
“One key finding from our trials of doing on policy, multi-teacher on policy distillation is that teacher models trained with substantially different training pipelines cannot be effectively combined through a straightforward on policy distillation merge, resulting in suboptimal performance.”
