Assigning roles to several coding agents does not establish that they actually follow that organization. OpenCollab is a framework for defining collaboration arrangements, running them on shared infrastructure and recording detailed execution events. Its Adherence measure asks whether the declared arrangement occurred in practice.
The authors report adherence ranging from 47.2% to 97.2% across configurations. That variation matters when interpreting a successful AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → team: the result may have come from behavior different from the workflow its designer intended. Recording actual collaboration gives researchers a way to investigate what the organization contributed.
Key takeaways · AI-distilled
OpenCollab argues multi-agentUsing several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.Full definition → coding evaluations typically assume agents faithfully follow their configured organization, and defines an Adherence metric that measures whether the declared organization is actually realized at runtime.
Changing any single dimension of the configuration shifted Adherence from 47.2% to as high as 97.2%, the authors report, so agents often collaborate quite differently from how they were set up.
The framework runs every configuration on a shared, controlled runtime and records fine-grained event streams, so gains can be attributed to the organization rather than to differences in system components.
A two-coder workflow built on OpenCollab reportedly set new state-of-the-art results against Mini-SWE-agent, Codex CLI and Claude Code, while its single-agent configuration used the fewest tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → across the evaluated suites. The paper is marked work in progress.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
articleAsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software EngineeringKaituo Zhang, Zhen Xiong, Zhimeng Jiang, Mingyu Zhong, Zhouyuan Yuan, Zhecheng Li, Bowen Lin, Chia-Yuan Chang, Mingzhi Hu, Huazheng Wang, Ying Lin
articleWorld of SWECraft: Agent Collaboration at Scale via Simulated Software WorldsShannon Zejiang Shen, Yapei Chang, Qiyao Liang, Benjamin Newman, Jenny Liang, Rulin Shao, Ziqiao Ma, Junjie Oscar Yin, Yung-Sung Chuang, Weijia Shi, Yejin Choi, Luke Zettlemoyer, Pang Wei Koh, Mohit Iyyer, Tongshuang Wu, David Sontag
articleWhat Open-Source Projects Tell Their Coding Agents: A Census of AGENTS.md and CLAUDE.mdPawel Zarzycki