5 useful things you'll learn in my new post-training textbook (shipping now!)

Nathan Lambert's post-training textbook treats reinforcement learning as an engineering problem as well as an algorithmic one. In the launch essay, Lambert describes systems that separate workers generating training examples from workers updating the model. Keeping those workers productive has to be balanced against how far the example-generating model has fallen behind the model being trained. That trade-off helps explain why understanding the update equation alone is insufficient for building a training system. Author's account

The book is aimed at readers with roughly undergraduate computer-science preparation. Lambert says it is freely available online alongside a 12-hour course, offering a route from foundational explanations to experimenting with post-training systems. Audience and accompanying material

“By word or page count, the book is about 25% RL. This seems appropriate.”

Nathan Lambert

“This intuition, from the policy-gradient theorem to PPO to modern versions like GSPO and CISPO, are crucial to understanding if a new algorithm is fake or has potential (no new algorithm will be proven right out of the gates).”

Nathan Lambert

“The core systems design, asynchronous RL with separate GPUs for the learners (the GPUs which take gradient steps) and actors (the GPUs which generate the rollouts in the environment), has been similar for a few years.”

Nathan Lambert

“The book will walk you through 3 eras, when researchers learned to do RL on preferences generally until ~2018, spent a few years learning how to apply it to language models from 2019 to 2022, and from 2023 on exploited the examples set by ChatGPT.”

Nathan Lambert

“With the pace of progress in AI, the time it takes for a research paper to land in a frontier model is 3-9months.”

Nathan Lambert