The Magic of LLM Distillation — Rishabh Agarwal, Google DeepMind
Source
youtube.com
Author
Latent Space
Date
Why it matters
Covers how modern distillationTraining a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.Full definition → differs from the 2015 Hinton approach, informing decisions on training smaller models from larger ones.
Terms in this piece · Glossary
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
