5 useful things you'll learn in my new post-training textbook (shipping now!)
- Source
- Nathan Lambert
- Author
- Nathan Lambert
- Date
- RLHF — Reinforcement learning from human feedback — training a model to prefer answers humans rate as better, which turns a raw text predictor into a usable assistant.
Post-training methods are the least-documented part of the model stack; a consolidated reference from a practitioner who trains open models is a durable resource, not a news blip.
“By word or page count, the book is about 25% RL. This seems appropriate.”
Nathan Lambert
“This intuition, from the policy-gradient theorem to PPO to modern versions like GSPO and CISPO, are crucial to understanding if a new algorithm is fake or has potential (no new algorithm will be proven right out of the gates).”
Nathan Lambert
“The core systems design, asynchronous RL with separate GPUs for the learners (the GPUs which take gradient steps) and actors (the GPUs which generate the rollouts in the environment), has been similar for a few years.”
Nathan Lambert
“The book will walk you through 3 eras, when researchers learned to do RL on preferences generally until ~2018, spent a few years learning how to apply it to language models from 2019 to 2022, and from 2023 on exploited the examples set by ChatGPT.”
Nathan Lambert
“With the pace of progress in AI, the time it takes for a research paper to land in a frontier model is 3-9months.”
Nathan Lambert
Checking sign-in…
Loading comments…