Vibeleaderboard
← All Intel
Intel / video

The Magic of LLM Distillation — Rishabh Agarwal, Google DeepMind

Source
youtube.com
Author
Latent Space
Date

Context

Rishabh Agarwal, a staff research scientist at Google DeepMind, walks through in a Latent Space interview. Distillation transfers capability from a costly teacher model to a smaller student. Agarwal gives the reason as cost against performance: a much better model that costs far more may go unused, and some uses, such as phones, cannot fit a large model. A co- adds that some models are now so big they exist mainly to serve as teachers, not to be run for users.

Agarwal contrasts that with the classic approach from Hinton's 2015 paper, where the student is trained to match the teacher's probabilities over every possible next word instead of one correct word. This is a generalized version of next-token prediction using soft labels in place of single right answers, and Agarwal says it has been used for . Agarwal also says training a large model and distilling it can beat training the small model directly, an effect people have seen repeatedly, though there are many competing theories.

Terms in this piece · Glossary
  • Large language model — A model trained on large amounts of data to process and generate language.
  • distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
  • Host — The computer on which a process runs.
  • pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
Read the source www.youtube.com
More from Latent Space
Recommended reads
Comments

Checking sign-in…

Loading comments…