← All IntelClip / AI AgentsNew benchmarks attempting to measure long-horizon code maintainability
From Harness Engineering is not Enough: Why Software Factories Fail — Dex Horthy, HumanLayer · ≈13:56
“there's a really cool one called Sweep Marathon from Abundant AI where they do like 400-hour tasks of like clone all of Microsoft Excel, every single feature.”
“Deep Sweep from Data Curve is also like large tasks on OSS repos that are not actually in the training set cuz they were never actually built in the real world.”
“if the model knew what good code looks like, it would probably write it in the first place.”
What’s in it
- Surveys next-gen coding benchmarks testing multi-hour, multi-PR agent tasks
- Names specific evals: Sweep Marathon, Deep Sweep, Frontier Code
- Explains the core limit of using AI judges to grade code quality
Clip transcript
look at these as like what is the future of evaluating code maintainability. Um there's a really cool one called Sweep Marathon from Abundant AI where they do like 400-hour tasks of like clone all of Microsoft Excel, every single feature. Uh and they have some sophisticated reward channel stuff. Uh Deep Sweep from Data Curve is also like large tasks on OSS repos that are not actually in the training set cuz they were never actually built in the real world. Uh and then you have Frontier Code from Cognition, um which is multi-PR tasks. They do interesting things like hey, if the model writes tests that don't fail on the pre-patch code, then it gets penalized. And then we have a judge model that says, "Okay, uh did this follow all of our code quality rules?" Um so we're getting better, but I think models judging quality can only go so far uh cuz if the new model if the model knew what good code looks like, it would probably write it in the first place.
Recommended reads
- clipWhy current coding benchmarks miss the real jobAI Engineer
articleLoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding AgentHan Li, Zhemin Fang, Rili Feng, Yingqi Zhao, Jiaheng Liu, Pengfei Gao, He Ye, Dayi Lin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang
videoThe Good, the Bad, and the Ugly: Why Coding Benchmarks Are BrokenAI Engineer
Comments
Checking sign-in…
Loading comments…