Hugging Face Journal Club: Direct On-Policy Distillation
- Source
- youtube.com
- Author
- Hugging Face
- Date
Context
Reinforcement learning (RL) improves a by rewarding good outputs, but running it on a large model is expensive. The paper Weak-to-Strong Generalization via Direct On-Policy Distillation proposes running RL only on a small model and transferring what RL changed. Instead of having the large model imitate the tuned small model's answers, the method compares the small model after RL with the same model before RL, and uses that difference in how likely each choice is as a dense, step-by-step reward for the larger model.
A Hugging Face journal club video describes the result as matching or beating direct RL on the larger model at a fraction of the compute. For one reported , the paper gives a transfer step of about four hours on eight A100 GPUs. The paper's conclusion also says the method can fail when the small model's improvement is not meaningful on the states the larger model actually visits. This account rests on the paper page and the video description, not a review of the full experiments.
- Model — A computational system that uses learned patterns to produce an output from information supplied to it.
- distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
- Training — The process that adjusts a model using examples and a measure of how well it performs.
- Setup — The combination of a model, a harness, its connections and the path the task's data takes, considered together for a task.
Checking sign-in…
Loading comments…





