Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Source
arxiv.org
Author
Rafael Rafailov et al.
Date
Context
Direct Preference Optimization simplifies training a LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → from comparisons between better and worse responses. The 2023 paper contrasts it with a common human-feedback pipeline: first train a separate model to score responses, then use reinforcement learning to improve those scores. DPO reformulates that objective as a direct classification-style training loss on preference data. This removes the separate reward-model stage and the need to sample fresh model responses during fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition →. In the reported experiments, it matched or improved response quality on summarization and single-turn dialogue while simplifying training.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
RLHF — Reinforcement learning from human feedback — training a model to prefer answers humans rate as better, which turns a raw text predictor into a usable assistant.