Brood War Bench: frontier agents play StarCraft at beginner level

Brood War Bench puts AI agents in control of StarCraft: Brood War, a real-time strategy game, and its report finds that none of them played beyond a beginner level. The author writes that a beginner playing a photon rush would have won every one of these games. The report's central observation is about time: the game keeps running while an agent thinks, and older models tended to play it as if it were turn-based, so they were destroyed while still deliberating.

The setup was a round robin among 19 model and effort configurations, each playing every other once, so every record covers 18 matches, a few of which ended at the time limit. Codex Astra at its xhigh effort setting went 18-0, and Astra at medium was second at 16-2. Claude Fable placed third at 15-3 and, apart from Astra at xhigh, was the only system to beat Astra at medium. At the bottom, Claude Haiku and Grok 4.6 at low effort won nothing, and the three Grok configurations won three games between them.

The author's game notes show how thinking time decided matches. Codex agents often sent a single Probe across the map to attack workers or buildings, and opponents spent dozens of seconds deciding how to deal with it instead of doing anything else. Grok 4.6 showed the extreme case: in one 43-minute game the xhigh run logged 11,138 reasoning tokens, issued only six batches of commands and never fielded a combat unit. More effort did not reliably help. Codex 5.6 Sol at medium went 13-5 while the same model at xhigh went 11-7. The author suggests newer models sometimes fell into the same slow-thinking trap, which may explain why some lower-effort settings did better. Codex also tended to split economy, production and army control among subagents that rarely communicated, so units went into attacks one at a time, a mistake the author says Codex handled much better in games where the author helped direct it.

The ranking measures these agents against one another, not against people, and each pairing was decided by a single game, so neighbors separated by a win or two are not firmly ordered. The extremes are the solid result: Astra at xhigh was unbeaten, and the Grok configurations barely managed to act. The explanations for why games went the way they did are the author's reading from watching them, not tested causes.