← Glossary
Glossary · training
distillation
Also: distilled
Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
A large "teacher" model generates high-quality outputs; a smaller "student" trains on them. The student learns from the teacher's rich, already-digested behavior rather than raw internet text, so it punches far above its size — most impressive small models are distillations of something bigger.
It's also a competitive fault line: API access to a frontier model is enough to distill from it, which providers' terms of service typically prohibit and which several public disputes have centered on.
In the index
- The Distillation Panic
- Training Agents 2: Live tutorial on model distillation for training custom agents.
- Introducing Arcee Supernova Medius A 14b Model That Rivals A 70b 2
- The Magic of LLM Distillation — Rishabh Agarwal, Google DeepMind
- The AI Frontier: from Gemini 3 Deep Think distilling to Flash — Jeff Dean