SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
Why it matters
Quantifies a real gap: agents that ace SWE-benchThe standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.Full definition → and Terminal-Bench perform far worse on legacy enterprise languages, especially where failures are silent, a concrete signal for where agentic coding still needs work.
articleLegacy-BenchFactory
articleLoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding AgentHan Li, Zhemin Fang, Rili Feng, Yingqi Zhao, Jiaheng Liu, Pengfei Gao, He Ye, Dayi Lin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang