← All IntelClip / AI AgentsBuild the eval first, then welcome all models
From Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute · ≈11:17
“if you want to build an agentic system start with the aval that matters and your ability to grade the outcome and then say I welcome all models”
Satya Nadella
“as soon as you have an aval the power is now in your hands. You can consider every single model. You don't have to trust brand. You don't have to trust somebody else's aval.”
“if you're doing something on a computer, then you should be seeing if you can use AI to automate part or all of that process that you're currently performing on a computer”
“I think ramp recently announced like rampbench which is aubench built off of their internal codebase and what that allows them to do is pick the coding agent or the model that performs best on their internal use cases”
“if you can make an aval to see how well agents use your product and then iterate on your product to make it more usable for agents, you're going to get more usage and therefore your product uh will become more valuable to agents”
What’s in it
- Argues (via Nadella) that owning your own eval flips the power dynamic: you can trade off cost vs. performance across all models instead of trusting vendors' public benchmarks.
Clip transcript
allows you to maximize data velocity and just increases progress in the industry. Uh so who needs a val? Uh now that we understand how to make them, we understand how to use them. Well, now who should actually be doing this? And the answer is every single company that uses computers. And I think that's probably close to all of the companies in the world. And the reason is because if you're doing something on a computer, then you should be seeing if you can use AI to automate part or all of that process that you're currently performing on a computer. Uh because that will increase the productivity of your company and therefore increase the value that your company generates. Uh so I like this quote from Satcha Nadella from the applied compute podcast he did last week. He says if you want to build an agentic system start with the aval that matters and your ability to grade the outcome and then say I welcome all models. So I like that last line because what he goes to say is that as soon as you have an aval the power is now in your hands. You can consider every single model. You don't have to trust brand. You don't have to trust somebody else's aval. You don't have to trust a public aval. Um and you can kind of skate the paro however you desire to uh balance that cost performance tradeoff. So step one build the eval step two kind of optimize against it. Um so what will people evaluate and uh I'm going to list four things and these four things are based off of what we actually see people using harbor to build a val for right now. Uh so so the first type of a val is people evaluating how well agents build their products. So everybody that's building a software product has some internal codebase or set of internal code bases. Um I think ramp recently announced like rampbench which is aubench built off of their internal codebase and what that allows them to do is pick the coding agent or the model that performs best on their internal use cases and they don't have to you know maximize token spend. instead they can make informed and educated decisions about how they build their products with agents. The second type of AVL that we see people build is how to evaluate uh how well agents use your product. So anybody that builds a software product is probably moving towards a world where they offer some sort of headless mode. So you see some companies have obvious have always been this way. They've been developer first like uh HubSpot and Stripe and things like that. And the idea is uh if you can make an aval to see how well agents use your product and then iterate on your product to make it more usable for agents, you're going to get more usage and therefore your product uh will become more valuable to agents. And then uh three evaluate how well agents power product features and then four evaluate how well agents automate internal processes. So depending on what type of company company you are, uh one or more of these might apply to you, but everybody should be considering right now how they can do one of these things. Um okay, so what
Comments
Sign in to comment.
Loading comments…