← All IntelClip / AI ToolsOpen Thoughts Agents: RL environment curation and SFT vs RL tradeoffs
From Data and Environment Curation for Post-Training LLMs — Mahesh Sathiamoorthy, Bespoke Labs · ≈11:55
“stronger models are not necessarily the uh best teachers right”
“SFT still contributed a lot to the gains”
“in many of the situations for example in enterprises SFT actually works works pretty well, right?”
What’s in it
- Explains why smaller teacher models beat Claude when training agents
- Shows SFT still drives most performance gains before RL adds marginal value
- Flags which curation tricks (synthetic rewriting, task augmentation) quietly failed
Clip transcript
working very well. And after the open thoughts work which was around uh data curation for reasoning models such as you know uh deepse kind of models, we moved on to open thoughts agents which is um very similar but how do you curate these the the data and RL environments for uh training agents now right not models. We we have a very similar figure here. Again we want to establish scaling loss. Um so as you increase the data set size we want to make sure that the curation recipe actually works. Uh and again I'm I'm not going to go into details here but very similarly there are various ways of choosing different sources for example stack exchange and and whatnot. How do you mix the test? How do you filter? Generating the rollouts uh choosing the teacher and so on. And again these are some of the lessons learnings u as an example even here we saw that stronger models are not necessarily the uh best teachers right so we found out some some some of the I think uh um quen models were better than for example um um um claude models I think and sampling multiple answers again helped in this case synthetic rewriting and task augmentation um is something we thought will work but it didn't very work work very well and the other thing is like in in this whole process of building this open thoughts agent SFT still contributed a lot to the gains um RL was kind of you know it's very comput inensive and for for the last few few percentages it really helped uh but but you know in many of the situations for example in enterprises SFT actually works works pretty well, right?
Comments
Sign in to comment.
Loading comments…