⚡️Open Questions in Agentic RL — Will Brown (Prime Intellect)
Source
youtube.com
Author
Latent Space
Date
Context
Training an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → on a long task creates a problem that a final right-or-wrong score cannot easily resolve: which earlier actions helped? This talk on reinforcement learning describes agents that search, run code and use other tools before producing an answer. As those sequences grow, a reward for the final result gives less direct guidance about each intervening decision. The speaker proposes evaluating intermediate steps too, such as whether a search helped the task. That creates another research challenge: building reliable judges for useful partial progress, especially where correctness cannot be checked as easily as a mathematical answer.
Key takeaways · AI-distilled
Brown argues models do not need to generate images in their reasoning. Programmatic tools such as crop, zoom and rotate are more efficient, and he recommends treating modes of intelligence as tool calls, which also makes them trainable.
Brown proposes using other models as tools, such as a subagentA separate agent spawned by another to do one scoped piece of work in its own context, returning only the result.Full definition → that answers one question about one website, to keep full pages out of the main context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →. Off-the-shelf small models need no training; the main model learns to prompt them.
For tasks with around 100 tool calls, Brown says turn-level rewards are needed, possibly from reasoning reward models that break tasks into rubrics. Scoring one turn could take a minute, which pushes training toward async pipelines that learn from stale turns.
Brown says model merging works surprisingly well: models trained separately on code, math or Spanish can have their weights averaged, which Cohere used extensively for Command A. He sees this as a path to learning skills in a decentralized way.
Brown, who did five years of multi-agentUsing several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.Full definition → RL in grad school, says group-chat agent setups like CrewAI mostly fail because models are never trained for them. He also notes that discarding thinking tokens between steps breaks current multi-turn RL math.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
subagent — A separate agent spawned by another to do one scoped piece of work in its own context, returning only the result.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
