Training Agents 3: Reinforcement Learning

This third session of a Hugging Face live series on training agents covers GRPO, group relative policy optimization, the reinforcement learning method in Hugging Face's TRL library. The useful idea is in the name: the model is never told how good an answer is in absolute terms. For each prompt it produces several attempts, a reward function scores them, and each attempt is pushed up or down by how far its score sits above or below the group's average, divided by the group's spread. The TRL documentation states the same formula, and calls the number of attempts per prompt num_generations.

That design explains most of the practical advice in the stream. Ben, presenting the theory, points out that if every attempt in a group succeeds, or every one fails, there is no difference to learn from, and a single success among many failures gives very little. He compares it to chess: always playing grandmasters or always playing beginners teaches you nothing. So the things you actually adjust are the size of the group, how hard the prompts are, and the reward itself, for example adding an easy format reward alongside a harder correctness reward so a struggling model still gets some signal. Compared with the distillation covered in part 2, which gives feedback at every token, this reward arrives once at the end of an attempt, and Ben describes typical pipelines as using distillation-style supervised training first and RL after it, sometimes cycling between them, rather than treating them as rivals.

Sergio's experiments in the stream make the point concrete. Scoring Python solutions by how many unit tests pass, a quick check before training gave a small model of about 0.6 billion parameters a mean reward of 0.22, which they read as hard enough to leave room to learn but not hopeless; they then trained it with groups of 2, 8 and 16 attempts. A third run used a deliberately gameable reward that paid out for writing Python code blocks: training reward rose to its ceiling while reward on the held-out test split did not. These are live demonstration runs, not benchmarks, but they show the failure the presenters warn about most, reward hacking, where the score climbs without the model learning the task.

Their answer is less a setting than a habit. Ben says the KL penalty, which limits how far the model drifts from where it started, will discourage some hacks but is not a fix; the real defence is a reward function that cannot be gamed, a dry run of that function on sample text, a short smoke test that shows at least some reward variation, and reading actual trajectories rather than admiring a rising curve. The next session, part 4, keeps GRPO but swaps simple reward functions for full environments. The full stream is on YouTube.