SFT, DPO or online RL: choosing a post-training method by its data
- Source
- x.com
- Date
New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by @BanghuaZ, Assistant Professor at the University of Washington @UW, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a…
Context
Choosing a post-training method depends partly on what feedback a team can provide. The Post-training of LLMs announcement distinguishes supervised , which learns from inputs paired with desired answers, from preference optimization, which learns to favor one answer over a rejected alternative.
Online reinforcement learning instead generates answers during training and scores them using human or automated feedback. This changes the preparation work from supplying answers or comparisons to defining useful rewards. The GRPO course explores that reward-design task in more detail.
- fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Checking sign-in…
Loading comments…