Vibeleaderboard
← All Intel
Intel / video

Hugging Face Journal Club: Direct On-Policy Distillation

Source
youtube.com
Author
Hugging Face
Date

Context

Reinforcement learning (RL) improves a by rewarding good outputs, but running it on a large model is expensive. The paper Weak-to-Strong Generalization via Direct On-Policy Distillation proposes running RL only on a small model and transferring what RL changed. Instead of having the large model imitate the tuned small model's answers, the method compares the small model after RL with the same model before RL, and uses that difference in how likely each choice is as a dense, step-by-step reward for the larger model.

A Hugging Face journal club video describes the result as matching or beating direct RL on the larger model at a fraction of the compute. For one reported , the paper gives a transfer step of about four hours on eight A100 GPUs. The paper's conclusion also says the method can fail when the small model's improvement is not meaningful on the states the larger model actually visits. This account rests on the paper page and the video description, not a review of the full experiments.

Terms in this piece · Glossary
  • Model — A computational system that uses learned patterns to produce an output from information supplied to it.
  • distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
  • Training — The process that adjusts a model using examples and a measure of how well it performs.
  • Setup — The combination of a model, a harness, its connections and the path the task's data takes, considered together for a task.
Read the source www.youtube.com
More from Hugging Face
Recommended reads
Comments

Checking sign-in…

Loading comments…