← All IntelClip / EducationGeneral Agent loop and supervised world-model signal beyond RL
From RL Without Verifiable Rewards (Will Brown, Prime Intellect) · ≈16:59
“beyond just RL, we found that it's quite important to think about cases where there's information in the world that RL alone will not explore.”
“there's this great work Echo from some researchers that we are friends with and have been collaborating with.”
What’s in it
- Explains a training loop that generates, solves, and synthesizes new agent tasks
- Shows how gating on pass rate boosts tool-use benchmark performance
- Argues pure RL misses world info that supervised learning from environments captures
Clip transcript
these are all the ways that we kind of gain confidence in the environments. I know we're running a little short on time, so I wanted to break up a couple blogs that we've put out recently that are kind of demonstrating pieces of this. We have a blog called general agent which is demonstrating this for tool use, this online loop of generating, solving, and synthesizing new tasks and gating based on this pass rate which then we train on and we see a great uplift on popular benchmarks for tool use. Um additionally, beyond just RL, we found that it's quite important to think about cases where there's information in the world that RL alone will not explore. And so, there's this great work Echo from some researchers that we are friends with and have been collaborating with. And then we did our own kind of deep dive into this as well to look at what happens when you have an agent that is not just training with reinforcement learning, but is also getting supervised learning signal from the environment itself. Which then allows the model to understand things like having a native world model of the environment, understanding what to expect because it has a likelihood model of the tokens that the environment itself will generate. And these are the sorts of things that in many cases allow the model itself to kind of more adaptively navigate the world and not just like refine its skill, but like get new information into its weights over time
Comments
Sign in to comment.
Loading comments…