Mirrorcode Preliminary Results

MirrorCode, a benchmark Epoch AI co-developed with METR, gives early evidence that AI agents can finish some coding tasks that would take a human engineer weeks, as long as the target is precisely specified and automatically checkable (epoch.ai). Each task asks an agent to rebuild an existing command-line program so that it produces exactly the same output. The agent can run the original program as a black box and read its documentation, but it never sees the source code and has no internet access, so it has to design the whole program itself. Grading uses hundreds to thousands of end-to-end tests, some held back as near-duplicates of visible ones so that hard-coding answers would fail.

The headline case is gotree, a bioinformatics toolkit of about 16,000 lines of Go with 40-plus commands for parsing and manipulating evolutionary trees. Running fully autonomously, Anthropic's Claude Opus 4.6 passed 2,000 of 2,001 tests when rebuilding it in Python, using 280 million tokens. Four researchers and engineers estimated the same job would take a skilled human without AI 2 to 17 weeks. Earlier Opus generations show how recent this is: on the same task the best of three runs scored 15% for Opus 4, 24% for Opus 4.1 and 63% for Opus 4.5. The authors say older models often submitted early, and the older two picked a tree data structure poorly suited to the problem. The largest program tested, the roughly 61,000-line Pkl configuration language, was not solved within a one-billion-token budget (about $550), though scores were still rising when the run ended.

These are preliminary results, limited to Claude models, with a follow-up release covering more programs and models; the authors say other models looked comparable or weaker in early testing. They flag two caveats that shape how far the result reaches. An existing program is an unusually exact specification, which is not how most software gets built, and they say this setup is the likely reason time horizons on MirrorCode look much longer than in other domains; for comparison, they cite METR's estimate of roughly 12 hours for Opus 4.6 on bug-fixing and research-engineering tasks. They also cannot fully rule out that models had seen these public codebases in training: they screened out a program that models could partly reproduce from memory, and the retained programs scored close to a baseline of code the models could not have seen.