SDPO Trains Models With Natural-Language Feedback Instead of Scalar Rewards

Most reinforcement learning for language models reduces feedback, whether from a grader, a compiler, or a critique, to a single reward number before training on it. Stanford SAIL's announcement points to the SDPO paper, which instead has the model condition on the feedback text itself, predict what it should have said given that feedback, and then distills those feedback-informed predictions back into the same model without the feedback present, using no separate reward model.

The authors report that across scientific reasoning, tool use, and competitive programming on LiveCodeBench v6, SDPO beats a strong GRPO baseline, 70.2% versus 66.6% aggregate accuracy, while reaching that accuracy with generations up to 11 times shorter. These are the paper's own reported comparisons against one baseline method, not a survey of all RL training approaches.