← All IntelClip / EducationThe economics of building agentic coding benchmarks
From Benchmaxxing: The Gap Between Benchmark Scores and Reality · ≈3:19
“You can't push the frontier forward from within the frontier.”
“If you try to use cheap labor, you're going to get what you pay for and the whole result is not going to be that useful.”
“We are not trying to minimize cost. We are trying to maximize quality.”
What’s in it
- Breaks down why building a 1,000-task coding benchmark costs $15M+
- Explains why AI-assisted labor can't fix benchmark quality problems
- Shares Surge's contrarian bet: pay top dollar, not minimum cost
Clip transcript
There are a handful of key antiatterns that we're going to go through. The first is price. Let's say you want to make an agentic coding benchmark, which these days is a very popular thing to want to do, and you want a thousand tasks in your benchmark. Each task takes 60 hours to make. Each software engineer in your workforce costs half a million a year. That's $15 million to make your benchmark. And if you think that over time about a third of those tasks are going to get washed away every year due to models getting better, that's $5 million to replace them. So that puts you out of budget for most projects. So then people turn to a variety of workarounds that have their own problems. One of which is trying to use a lot of AI assistance which ultimately does not really work. Like you can't push the frontier forward from within the frontier. You need to inject that external human expertise and it needs to be good expertise. If you try to use cheap labor, you're going to get what you pay for and the whole result is not going to be that useful. At Surge, one of our differentiators has long been that we are not trying to minimize cost. We are trying to maximize quality and part of that means paying a lot of money for good workers. We've always believed that but especially in 2026 models are just beyond the point where you can make do with anything less than the best workers.
Comments
Sign in to comment.
Loading comments…