Brood War Bench
- Source
- bw.swerdlow.dev
- Author
- benswerd
- Date
Ben Swerdlow built an agent-only, playable version of StarCraft: Brood War and ran 171 matches pitting Codex, Claude, and Grok models against each other in real time. Real-time strategy is a rare agent benchmark that stresses different muscles than a single-turn coding or reasoning task: continuous decision-making under a clock, resource management, and multi-unit control all at once. The write-up reports that Codex's disruption-focused tactics beat sustained macro play, that Grok models tended to over-deliberate and never actually field units, and that subagent architectures without communication between the parts broke down during coordination-heavy moments. The result is a small but concrete catalog of failure modes for multi-agent control under time pressure, useful for anyone designing agent harnesses meant to act continuously rather than respond once per turn.
Context
Brood War Bench tests whether AI agents can keep acting while the real-time strategy game StarCraft: Brood War continues around them. Its report describes a round-robin competition between model and reasoning-effort configurations. The author observed agents losing games while thinking, one model struggling to sustain production, and subagents sending units into attacks without assembling a coordinated force.
These failures make response timing part of the task itself: an must repeatedly observe, decide and act before conditions change. The author judged every tested agent to be at beginner level, including the strongest competitors. Winning this tournament therefore measures relative performance among the tested systems, while the observed failures reveal how much real-time coordination remains unsolved.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Checking sign-in…
Loading comments…