← All IntelClip / EducationWorld simulators for uncontrollable environments (web/tools)
From RL Without Verifiable Rewards (Will Brown, Prime Intellect) · ≈12:06
“we found that these simulators are actually really great for RL because you can one, if you have production data, you can make your simulator better and better over time, but also you have full controllability over the back-end”
“you can actually do this reverse engineering where you get to kind of plant the answer”
“you don't know in advance if a task was solvable at the time that you are being asked it”
What’s in it
- Explains how to build simulators for tools you can't directly control
- Shows a technique to bake solvability guarantees into RL training data
- Covers reverse-engineering tasks by planting the answer and working backward
Clip transcript
again uh can be applied more generally beyond code as well. Uh and so, we talk about world simulators broadly as the the sort of thing we might want to do in messier environments which are not just uh production uh like which are not just doc search or code. And so, a lot of the ones that we've been working on at Prime and Elect are related to things like tool use and web applications where we don't actually have full controllability of the back-end state. There are some MCP tools or CLI tools or websites or applications where we can't actually program them yet. And so, what we want to do is learn to simulate them. And so, we found that using combinations of uh universal back-end infrastructure and test time scaling and search and kind of iterating between the simulator and the real uh and the real, we can kind of ground in these production traces. This then allows us to create really high-fidelity simulators. And so, we found that these simulators are actually really great for RL because you can one, if you have production data, you can make your simulator better and better over time, but also you have full controllability over the back-end. And so, you can actually do this reverse engineering where you get to kind of plant the answer. You can start from the end and work backwards uh and so that you you have this verifiability baked into the simulator even if you don't have it in the real-world production deployment because you don't know in advance if a task was solvable at the time that you are being asked it. Um
Comments
Sign in to comment.
Loading comments…