Clip transcript
>> Uh oh yeah, they absolutely do use more tokens. I I actually I want to kind of like riff on something that Tuhin was saying, which is like you know, a lot of times when you look at these different benchmarks, you'll see that the small models will perform better than like even the frontier models in certain cases. I think a lot of people they look at this and they immediately jump to, "Oh, how can we just route like, you know, the task where the small models do better just straight to the smaller models?" I I think that one of the things we really want to emphasize with our recent blog post and recent Devin Fusion was that this kind of like naive like initial routing to based on the task type is extremely fragile, especially the more agentic the task you you work on is. So, for example, like a real developer, you might ask your agent first, "Oh, how does this code base work?" And then you you go deeper and you're like, "Okay, actually, can you implement some features for me?" And then you go deeper and you're like, "Oh, can you like now go to like a live test of this feature and debug deep cases?" The complexity changes and the type of task changes over time, and you don't want to be left with some subpar model for the the task that you're now on. I think this is why people like frontier models so much is they're they're just like generally intelligent and they're capable of shifting between various different domains, even if you can eke out better performance in very specific tasks. Um, and and what So, the challenge is, how do you get a small model to know that it's out of its depth and you need to now like go switch to another model or go like, you know, go to a smart smarter model. And and our solution to this is you kind of just always have this like main frontier agent that's watching, even if it's not the one doing the work. It should at least be keeping tabs to figure out, "Okay, we like the the agent I delegated to now is like out of its depth. I need to kind of like move it to to something else." Um, and and overall, just the guarantee of always having frontier intelligence present, I think reduces the the fragility of of these systems