SWE-Duel: Towards a Self-Scaling Arena for Coding Agents
Source
arxiv.org
Author
Wei Minn, Zhensu Sun, Lwin Khin Shar, David Lo
Date
Context
SWE-Duel tests whether a coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → can discover a defect when visible tests still pass. One agent adds a feature to a real repository, hides a bug inside it and writes separate tests that expose the bug. Those revealing tests stay hidden from the reviewing agent.
A challenge is admitted only after a fresh session of its author repairs it without seeing the hidden evidence. The opponent must then fix the defect while passing the feature tests and the repository's existing tests. Simply deleting the new feature cannot win. This makes the benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → a test of reviewing apparently working code and preserving functionality, with challenges generated by the participating agents.
Key takeaways · AI-distilled
A Red pull request counts only if it passes its feature tests and the repo's existing tests while failing exactly the sealed bug tests, and a fresh session of the same agent can repair the bug blind, which filters out unsolvable tasks.
The tournament ran 11 model-agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → participants on 15 repositories in five languages for 1,500 turns.
Leaders win in different ways: GPT-5.5 with Codex CLI ranked first as solver, repairing 98.3% of defects it reviewed; Claude Opus 4.8 with mini-swe-agent ranked second as task author with a 27.3% Red win rate.
Within one model family, tasks written by the newest version beat a fixed reference solver 42.9% of the time versus 4.0% for older versions, which the authors call preliminary evidence that difficulty scales with authoring ability.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
articleCoaching Qwen3 Coder 30B to Think Like a CodeClash Arena AgentIvy Ning Zhang
articleCheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?Hang He, Li Wang, Hao Chen, Yuchen Shao, Yuling Shi, Lisheng Wang, Peiyang Liu, Goose Lin, Zaiyuan Wang, Haiying Sun, Ting Su, Chengcheng Wan
articleZero2Repo: Can Coding Agents Build Repositories from Scratch?Pei Yang, Tianyu Shi, Yuhang Yao, Wanyi Chen, Tongyun Yang, Dun Pei, Haonan Wang, Pengbin Feng, Guanxu Yu, Jingchun Huang, Zeyu Zhang, Shuhan Sun, Hao Li, Xiang Li, Jie Xiao, Xinyu Wang, Hanxin Chen, Daqi Li, Qi Jia, Hongshan Lin, Zhizhou Gu, Zijun Tian, Weizhi Du, Lynn Ai, Eric Yang