RL for Agents Workshop - Deep Dive on Training Agents with RL and Open Source

This Hugging Face workshop on training agents with reinforcement learning keeps coming back to one idea. The difficult part is describing the task well, and that description also serves as your test. Lewis from Hugging Face opens by explaining how the work changes. The RL recipe that most open-source work has used gives the model a prompt and has a checker return 1 or 0 for the answer. An agent instead acts over many steps inside an environment. It calls tools or runs commands, receives observations, and gets scored only when the episode ends, which can be hours later. He lists what an environment holds: the tasks, an execution backend such as a sandbox, browser or terminal, some state such as a database or files, and a way to assign reward. That reward is now often a mix of simple rules and rubrics graded by another LLM.

Will Brown of Prime Intellect goes further and says, by his own account somewhat provocatively, that evals and environments are the same thing. In his framing an environment bundles tasks (the dataset), a harness (the agent or tool interface the model runs inside) and metrics (reward functions in RL, scoring rules in an eval). Once it is built, the same object can tell you whether a prompt change, a cheaper model or a training run helped. His team's verifiers library describes itself as a way of creating environments to train and evaluate LLMs. Brown argues that the harness a model is trained in should match the one it will be deployed in. His answer to where the data comes from is your own usage: log the prompts you give your coding agent, then treat the diff that was finally merged, or the tests added with it, as the target.

The other talks explain why a good test is the scarce part. Ofir Press, from the SWE-bench team, says benchmarks usually saturate within a year or two, and he prefers answers that can be checked deterministically over LLM judges. Lewis warns that agent evals with terminal access invite reward hacking, meaning the agent scores well by exploiting a flaw in the test instead of doing the task, and that open models can look strong on public benchmarks but then underperform on newer ones. His advice is to keep evals of your own. Brown is less strict about judges. He trusts them for narrow yes-or-no questions that most informed people would agree on, averaged over many such criteria.

The panel adds a practical caveat. Asked what reliably works, Brown and Lewis both point first to distillation: plain supervised fine-tuning of a small model on a larger model's traces. Lewis notes that this is what most teams, especially in enterprises, already do, while Brown expects RL to become cheaper and easier to justify. An earlier session in Hugging Face's Training Agents series explains step by step why rewards end up inside environments, and a live distillation tutorial shows that simpler route in practice.