← All IntelClip / OtherWhy 80-99% accuracy matters more than 50% on capability graphs
From Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs · ≈7:46
“Now, typically this graph is shared with the 50% accuracy rate, but I think it's much better to actually look at the 80% accuracy rate or higher.”
“You don't want to be creating a plan or a spec and then have a 50/50 chance of coming back and knowing that you wasted compute and your attention span.”
“Now, if you're kicking off a process that is going to take an hour and it has a 50% chance of completing, there's a very high chance you just wasted that hour and you could have been doing something different.”
What’s in it
- Reframes the AI agent 'task-length' meter graph using higher accuracy thresholds
- Explains why 50% accuracy benchmarks overstate agent reliability
- Offers a mental model for deciding when it's safe to delegate a task to an agent
Clip transcript
or 5:00 p.m. I'm going to set off an agent and then come back." And then you realize there is a there is a critical flaw. Now, the reason I bring this up is it's very important to have a good mental model and understanding how accurate models can be in accomplishing tasks. Who here has seen this meter graph before? I think a decent number of people. So, this is pretty common on actually mapping how much time models can complete tasks of certain categories for. So, the idea being that as models get better and better, they can do longer-running tasks. Now, typically this graph is shared with the 50% accuracy rate, but I think it's much better to actually look at the 80% accuracy rate or higher. And you can see there you can still see a similar exponential trend, but we're no longer claiming that models can accomplish tasks that would take a human 18+ hours. Now, I actually think it's much better to measure the accuracy at 90% or 99% because this is where the mental model is most efficient. You construct a plan, you create a spec, [snorts] you hand it off to an agent, and you're pretty sure that it'll get things done, right? You don't want to be creating a plan or a spec and then have a 50/50 chance of coming back and knowing that you wasted compute and your attention span. Now, if you're kicking off a process that is going to take an hour and it has a 50% chance of completing, there's a very high chance you just wasted that hour and you could have been doing something different.
Comments
Sign in to comment.
Loading comments…