Training Agents 2: Live tutorial on model distillation for training custom agents.

This Hugging Face live stream, the second in a series on training agents, builds its explanation around the difference between copying a teacher and being graded by one. Ben, one of the two presenters, uses chess. If a small "student" model learns from complete games played by a grandmaster "teacher", it gets plenty of signal, but the games never cover the student's own beginner mistakes, such as leaving the king open. That is off-policy distillation: the student copies sequences that someone else produced. The supervised fine-tuning on a coding agent's traces in part 1 of the series is, in this stream's terms, exactly that.

On-policy distillation reverses who does the writing. The student answers each prompt itself, mistakes included. The frozen teacher then reads that answer once, without generating anything, and gives its own probability for every token. The loss is the gap between the two at each position. Ben's argument is that this combines the strengths of two other methods. Like supervised fine-tuning, the signal is dense, with feedback on every token. Like reinforcement learning, which part 3 covers, it is about what the student actually did. Reinforcement learning only gives a reward when the task ends, and Ben points out that a student who never makes a legal first move never gets a reward at all.

Ben also explains why the loss uses reverse KL divergence. He calls forward KL "mean seeking": it can pull the student toward an average of the teacher's options that matches no real move. Reverse KL is "mode seeking": it pushes the student toward the specific choices the teacher actually favours. He presents this as an intuition and says the way to really understand it is to train models and watch how they behave.

The practical point is that both methods live in one class in TRL, Hugging Face's training library. In the GKDTrainer docs, lmbda sets how much of the training data the student generates and beta moves the loss between forward and reverse KL. Sergio, the other presenter, ran off-policy with both set to 0 and on-policy with both set to 1. In each case he had a coding agent write the script and sweep three learning rates on Hugging Face's cloud jobs. He calls these illustrative runs that finish in minutes, not results. Ben adds a limit: a student trained with on-policy distillation cannot be expected to surpass its teacher, whereas reinforcement learning is limited by its environment and reward, which is where the series goes next.