E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
Source
arxiv.org
Author
Hantian Ding, Chloe Bi, Jiacheng Zhu, John Yang, Matt Deitke, Pengcheng Yin, Zijian Wang, Rui Hou
Date
Context
E2E-SWE asks whether a coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → can turn a written specification and an empty workspace into a complete, installable project. Its 186 tasks span eleven programming languages, and hidden tests check whether the resulting software implements the required behavior.
Across thirteen evaluated models, the authors report first-attempt success rates from 11.7% to 67.7%. This gives a different view of coding ability from fixing a localized defect: the agent must assemble a working project whose parts satisfy the specification together. The strongest reported result still leaves substantial room for improvement on these tasks.
Key takeaways · AI-distilled
E2E-SWE has 186 whole-repository tasks across 11 languages. Given only a natural-language spec and an empty workspace, the agent must build a complete, installable project that passes a comprehensive hidden test suite.
Each task is written by a software engineer working with an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → to produce the tests and an implementation-independent spec, then audited by autonomous agents that repair defects using static inspection and failures seen in real model rollouts.
Across 13 frontier models, pass@1 ranged from 11.7% to 67.7%, which the authors say separates models clearly while leaving considerable headroom.
Trajectory analysis found long, front-loaded reasoning, which the authors read as a sign that building a working codebase from scratch depends on planning and system-level reasoning up front.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleZero2Repo: Can Coding Agents Build Repositories from Scratch?Pei Yang, Tianyu Shi, Yuhang Yao, Wanyi Chen, Tongyun Yang, Dun Pei, Haonan Wang, Pengbin Feng, Guanxu Yu, Jingchun Huang, Zeyu Zhang, Shuhan Sun, Hao Li, Xiang Li, Jie Xiao, Xinyu Wang, Hanxin Chen, Daqi Li, Qi Jia, Hongshan Lin, Zhizhou Gu, Zijun Tian, Weizhi Du, Lynn Ai, Eric Yang
articleSWE-Duel: Towards a Self-Scaling Arena for Coding AgentsWei Minn, Zhensu Sun, Lwin Khin Shar, David Lo
articleAgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSLDipin Khati, Shubham Mehrotra, Bin Bi, Zhou Yu, Denys Poshyvanyk, James Zhu, Sitaram Asur, Phil L. Mui