← All IntelClip / EducationUsing production traces as raw training material
From RL Without Verifiable Rewards (Will Brown, Prime Intellect) · ≈9:43
“And so uh taking these sources, these raw materials, as places to kind of search for tasks from is a very useful way of starting to kind of create this environment out of nothing.”
“And because you have the real world, you want to use that, the real world, your production environment, your agent traces, as the source from which you want to learn, even if you don't have supervision yet.”
What’s in it
- Shows how to bootstrap agent training data from live traces
- Explains building RL environments before you have any labels
- Argues production logs beat synthetic data for grounding agents
Clip transcript
answering questions about the quality of a rollout. And so for source material, um one very useful version of this, especially for this continual learning goal, is production traces themselves. And so what we found is super helpful is taking existing traces from a deployed agent and treating these as the source material where we don't necessarily know up front what the distribution of tasks is, but as an agent is deployed, you start collecting more and more examples of let's say user prompts or uh uh calls from an orchestrator agent down into a sub-agent. And this starts becoming the distribution. We don't have labels yet, but it tells us at least what we want to look for. Um and so this is one very useful category, especially for other things like uh search or for code. Uh you have doc corpora corpora corpora of documents, you have repos that are also very useful for kind of anchoring your um your uh your learning as well. That are And so uh taking these sources, these raw materials, as places to kind of search for tasks from is a very useful way of starting to kind of create this environment out of nothing. Well, it's not nothing, it's something from the real world. And because you have the real world, you want to use that, the real world, your production environment, your agent traces, as the source from which you want to learn, even if you don't have supervision yet.
Comments
Sign in to comment.
Loading comments…