NVIDIA's PivotOPD trains agents to recover from early mistakes

When a language agent works through a task over many turns, one wrong action changes every state it sees afterwards, so errors compound. NVIDIA researchers, with Princeton and the University of Maryland, report that in preliminary experiments with three Qwen3 models, more than half of failed rollouts contained a pivotal mistake, an action that moves the agent farther from finishing, and it typically came early. Those mistakes were often recoverable: the narrated video attached to NVIDIA's post, describing ALFWorld, says correcting just the pivotal turn lifted success from 8 to 59 percent.

On-policy distillation trains a student by having a teacher score the student's own attempts token by token. The video explains why that misses this failure: overall failures drop, but failures after a pivotal mistake barely move, because the recovery action sits below 1 percent probability, so the student never samples it and never receives a signal for it. PivotOPD fixes the targeting. A teacher rereads each rollout in hindsight, names the correct action at the turn that went wrong and a recovery action for each of the next few turns. One training term pushes the student away from the original mistake; a second trains it on recovery responses written by a hinted copy of the student itself, so the targets stay in the student's own reasoning style.

In the project page's replay of 72 labelled pivotal mistakes, PivotOPD recovered 72.7 percent of the time, against 8.3 percent for the base model and 20.3 percent for standard distillation, and it worsened none of them. Against 13 baselines it had the best average on ALFWorld, WebShop and search-based question answering for both Qwen3-1.7B and Qwen3-8B students, and it stayed ahead when Qwen3-8B acted as its own teacher, which the authors read as showing that much of the gain comes from where and how the teacher intervenes rather than from a stronger teacher. On SWE-Bench Verified, a Nemotron-3.5 student rose from 62.8 to 66.0 percent resolved, where standard distillation added 0.2 points; that run used only the preventive term. All results are self-reported by the authors, and the full paper text was not part of the reviewed material.