Zero2Repo: Can Coding Agents Build Repositories from Scratch?
Source
arxiv.org
Author
Pei Yang, Tianyu Shi, Yuhang Yao, Wanyi Chen, Tongyun Yang, Dun Pei, Haonan Wang, Pengbin Feng, Guanxu Yu, Jingchun Huang, Zeyu Zhang, Shuhan Sun, Hao Li, Xiang Li, Jie Xiao, Xinyu Wang, Hanxin Chen, Daqi Li, Qi Jia, Hongshan Lin, Zhizhou Gu, Zijun Tian, Weizhi Du, Lynn Ai, Eric Yang
Date
Context
A coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → can implement almost an entire specification and still miss a requirement that makes the result incorrect. Zero2Repo tests this by giving agents requirements, interface rules and an empty workspace, then checking the submitted repository against hidden tests. A task passes only when every test succeeds.
In the authors' eleven-task evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →, the strongest agent completed ten tasks. Failed submissions nevertheless passed 90–99% of their tests, with failures for the two strongest agents concentrated in individual omissions or uncommon rules. That makes specification coverage a concrete weakness to investigate, even when the generated software looks largely complete.
Key takeaways · AI-distilled
Each Zero2Repo task gives the agent a product requirements document, an interface contract and an empty workspace. Tasks come from real, version-pinned open-source projects converted into specs, reproducible environments and hidden acceptance tests.
The hidden tests are validated both ways: a reference implementation derived from the upstream project must pass them, and adversarial validation must show they reject incorrect implementations. Scoring is binary, every test must pass, with no LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judge.
On 11 tasks drawn from repositories frontier models have very likely seen in training, the strongest agent solved 10, and every failing submission still passed 90-99% of the hidden tests, the authors report.
For the two strongest agents, 67-100% of failed tests traced to a single omission or a low-frequency rule stated in the spec, not a missing subsystem, so the authors frame each failure as a concrete target for improvement.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleSlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code RepositoriesMingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis
articleE2E-SWE: Benchmarking LLMs on Building Working Codebases from ScratchHantian Ding, Chloe Bi, Jiacheng Zhu, John Yang, Matt Deitke, Pengcheng Yin, Zijian Wang, Rui Hou
articleAcquiring and Verifying Repository Norms for Coding AgentsKaifeng He, Xiaojun Zhang, Zhenxi Chen, Christoph Treude, Mingwei Liu