Our latest post explores on-policy distillation, a training approach that unites the error-correcting relevance of RL with the reward density of SFT. When training it for math reasoning and as an internal chat assistant, we find that on-policy distillation can outperform other approaches for a fraction of the cost.
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Why it matters
On-policy distillationTraining a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.Full definition → combines RL's error correction with the dense reward signal of supervised fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition →, and Thinking Machines reports it beating other approaches on math reasoning and an internal assistant for a fraction of the training cost.
