Intel / article

AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering

Source: arxiv.org
Author: Kaituo Zhang, Zhen Xiong, Zhimeng Jiang, Mingyu Zhong, Zhouyuan Yuan, Zhecheng Li, Bowen Lin, Chia-Yuan Chang, Mingzhi Hu, Huazheng Wang, Ying Lin
Date: Published Sep 29, 2026

Context

AsynCodeBench tests whether coding agents' separately produced components actually work together. When one depends on another's function or data structure, it can keep working against an outdated version even while making progress on its own assignment. The benchmark makes these relationships explicit across 19 repository tasks and 52 dependencies between components. Executable checkers evaluate each dependency, and the authors count a dependency as satisfied only by its outcome in the integrated workspace. It records how many dependencies are satisfied and when they first become satisfied. The authors report that improvements in coding performance do not necessarily translate into stronger collaboration, giving teams a way to examine failed handoffs that a final task score can obscure.

Key takeaways · AI-distilled

AsynCodeBench scores coding on collaboration itself: each of its 19 tasks from real repositories carries an explicit dependency graph, 52 cross-agent dependencies in total, checked by executable Dependency Checkers.
The authors propose two metrics: Asynchronous Dependency Pass Rate, how many cross-agent dependencies end up satisfied, and Dependency Resolution Step, when each dependency is first satisfied during the run.
Across model families, scales and generations, better coding scores did not reliably mean better collaboration, and task-level results could diverge sharply from dependency-level measures, so task pass rates alone can hide coordination failures.
Trajectory analysis found coordination tends to succeed in bursts, many dependencies resolving within a short stretch of the run, a pattern the authors call a hopping window, rather than improving gradually.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.

Read the source arxiv.org