RL Without Verifiable Rewards (Will Brown, Prime Intellect)

Training an agent on messy work requires constructing feedback even when success lacks a simple pass-or-fail test. Will Brown of Prime Intellect describes starting from real documents, completed code changes and records of deployed agents to create tasks grounded in actual work.

One technique works backward from a known achievable result, removes the solution and asks the agent to reconstruct it. Model-based judges can then examine completed attempts and identify mistakes with hindsight. Brown presents these methods as steps toward agents that learn from experience, with human experts still deciding which outcomes matter and checking whether the scoring rewards the intended behavior.

“environments and evals are really the same thing”

Will Brown

“reward hacking is when you have a kind of loose proxy for your objective that is undefined at the boundaries”

Will Brown

“You can verify the easy problem and then learn on the hard problem.”

Will Brown

“RL's great for refining skills, but less so for incorporating like dense new knowledge.”

Will Brown

“classical machine learning will tell you you can train for the distribution, but generalizing outside of the distribution is kind of an undefined problem”

Will Brown