Scaling up Continual Learning — Ronak Malde, Trajectory

A model can supply its own training guidance when one copy receives a useful hint and another must solve the task without it. The self-distillation talk description explains that the unhinted student learns from the hinted teacher's predictions, providing feedback throughout an answer rather than only a final score.

Long tasks introduce a reported failure: the student's path can diverge so far that the teacher repeatedly corrects it, encouraging excessive hesitation. A hint containing the answer can also teach the student to state that answer and invent the reasoning afterward. The proposed remedies target these failures; the account does not establish that self-distillation is universally superior.

“Scale on policy self distillation to trajectories with a hundred tool calls and the model collapses into hedging. The tokens it learns to favor fill up with wait, but, and maybe, until, as Ronak Malde puts it, everything just turns into maybe.”

AI Engineer

“At the frontier there is no smarter model to distill from, so you make the model its own teacher: put privileged information, a hint, in the teacher's prompt, and match the log probs of the student that never saw it.”

AI Engineer

“GRPO gets on policy sampling but explodes parallelism and collapses feedback into one sequence level score, which he compares to being handed 87 out of 100 on an essay and told to work out why.”

AI Engineer

“Self distillation gets all four, and it optimizes across the entire vocabulary at every token instead of sharpening the one that was sampled, which is why it keeps climbing past where GRPO plateaus while tokens to solve go down rather than up.”

AI Engineer

“The failure modes are the useful part: step level KL weighting to handle divergence, and residual guidance for hint leakage, the self distillation analogue of reward hacking, where a hint containing the answer teaches the model to state it and back fill the reasoning afterward.”

AI Engineer