What It Takes For Coding Agents To Complete Large Software Tasks
- Source
- factory.com
- Author
- Factory News
- Date
Factory compared single-agent and multi-role systems rebuilding real programs, including gdal and 7-Zip, purely from observing their black-box behavior, with no access to the original source. Adding one specific role to the process, an agent whose job is to define a rigorous completion standard before implementation starts, raised gdal's behavioral parity with the original from 36 percent to 90 percent, with comparable jumps on other targets.
The failure mode this fixes is familiar to anyone running long-horizon coding agents: an agent declares a task done because it stopped finding new problems, not because it verified the result against an externally defined bar. Without a completion standard set in advance, done defaults to whatever the implementing agent happens to check, which tends to miss edge cases the original author handled. Splitting that judgment into a separate role, defined before implementation begins, gives the system an external check instead of letting the same agent grade its own work.
Context
Factory argues that a coding can stop before a large project is complete because its checks cover only what it thought to build. In a Factory-run experiment with its Droid agent, a separate validator defined behavioral checks before implementation, while an orchestrator directed repairs. For one recreation of the geospatial program GDAL, the reported hidden-test score rose from about 36% to 90%.
The comparison used one campaign per condition and substantially different resources: that system run consumed roughly fourteen times the credits and thirteen times the elapsed time. It supports independent completion checks as a promising design, without isolating their benefit at equal compute. The Missions architecture provides related on separating implementation from validation.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Checking sign-in…
Loading comments…