Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Source
arxiv.org
Author
Rafael Rafailov et al.
Date
Why it matters
It stripped most of the machinery out of preference training: no separate reward model, no RL loop, just a classification loss that reaches comparable quality and is stable enough to run without a research team. Most open-model post-training you see now is DPO or a descendant.
Terms in this piece · Glossary
RLHF — Reinforcement learning from human feedback — training a model to prefer answers humans rate as better, which turns a raw text predictor into a usable assistant.
