Intel / article

World of SWECraft: Agent Collaboration at Scale via Simulated Software Worlds

Source: arxiv.org
Author: Shannon Zejiang Shen, Yapei Chang, Qiyao Liang, Benjamin Newman, Jenny Liang, Rulin Shao, Ziqiao Ma, Junjie Oscar Yin, Yung-Sung Chuang, Weijia Shi, Yejin Choi, Luke Zettlemoyer, Pang Wei Koh, Mohit Iyyer, Tongshuang Wu, David Sontag
Date: Published Oct 20, 2026

Context

The World of SWECraft paper introduces Software World, a test environment for a question that grows as labs run more agents in parallel: when AI agents choose their own work on shared software, does collaboration produce better results? Each "world" is built from real Python packages that depend on one another, such as requests and urllib3, or numpy and pandas. Agents maintain those packages on a GitHub-like server with only a broad goal: make the software faster while keeping it correct. They decide what to work on and communicate through issues and pull requests. The scoring is designed to resist gaming. The authors run human-written benchmarks from other packages that depend on the world's libraries but that the agents never see. A change counts only if it speeds up that outside code.

Across three worlds, eight models and more than 10,000 hours of work, the best run in each world sped up the hidden packages by between 1.14 and 1.31 times, which is where the paper's "up to 31%" comes from. gemini-3.8-flash led in all three worlds. Several frontier models stopped early because they concluded nothing was left to do. claude-opus-5-5 and claude-fable-5-1 sat idle for more than 90% of their sessions, and gpt-6-astra had no recorded repository activity in 91% of its completed sessions. A prompt that urged exploration roughly doubled the Opus and Fable work sessions and modestly raised their scores, but it left Astra's score essentially unchanged.

The authors find that collaboration does not help automatically. Six Opus agents that each owned one package beat six that could edit any package, which sometimes duplicated each other's work. For gpt-5.6-sol and kimi-k2.7-code, however, one agent maintaining all six packages beat six agents. More communication did not reliably improve results, and agents with a shared budget left some of it unused. The study covers three Python ecosystems and one goal, software efficiency.

Key takeaways · AI-distilled

The authors build simulated software worlds out of real, interdependent Python packages, let agents pick their own tasks under a goal of improving efficiency, and score the group by measuring speedup in hidden dependent packages.
Communication, ownership, and resource allocation can each be varied, which the authors say lets them study separate elements of collaboration within the same framework.
Across three worlds, eight models, and more than 10,000 hours of simulated agent work, the authors report that performance rises with , the best model improving downstream efficiency by up to 31%.
Most agents located useful work through collaboration, but the authors report that some frontier models stopped looking for further improvements too early and made little progress.
The authors say they are releasing the simulation framework and the agent traces so that later studies of agent collaboration can build on them.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
test-time compute — Spending more computation when the model answers — thinking longer, trying multiple attempts — to buy accuracy without training a bigger model.

Read the source arxiv.org