On Policy Distillation
- Source
- thinkingmachines.ai
- Author
- Thinking Machines
- Date
Context
On-policy trains a model using responses that it generates itself, with a teacher model grading each generated text unit. Thinking Machines uses this feedback to guide the learner in situations it actually encounters, including its own mistakes. This provides more detailed guidance than a single score for the finished answer, which is the limitation of reinforcement learning that it describes. Method.
In one internal-assistant experiment, the starting model was Qwen3-8B, which scored 85% on an instruction-following test (IF-) and 18% on an internal-document question set. on a 70/30 mix of internal documents and chat data raised the question score to 36% but lowered instruction-following to 79%. A further phase of on-policy distillation, with the earlier Qwen3-8B as teacher on general chat prompts, brought those figures to 41% and 83%. This is a single experiment by the method's authors, and it supports using the method to recover much of the behavior lost after adding domain knowledge, not a guarantee that all of it returns. Personalization results.
- distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Checking sign-in…
Loading comments…