Building a Magic the Gathering Benchmark

A developer who writes as Sean has built a benchmark that tests AI models on Magic: The Gathering puzzles rather than full games, because he found that win rates mostly measured luck and the choice of decks. Magic involves a lot of chance and deck matchups work like rock-paper-scissors, so, he writes, "if I only looked at wins it would depend more on what decks were picked than which models were playing" (Sean's post). Each puzzle is a single turn with one line of play that wins, it takes at least 15 actions so random play cannot stumble into the answer, and the result is pass or fail.

He first had to fix the harness, the software that connects a model to the game. The models played through Forge, an open-source Magic rules engine, and he traced many of their early blunders to the harness rather than the models. It read decisions out of the model's prose, so a model saying it would keep its opening hand could trigger a mulligan, and it quietly let Forge's own AI make some choices, such as which lands to tap. He switched to tool calls with fixed fields, showed models the full game state with every legal action, gave each game object a unique ID, and removed the silent fallbacks.

Of more than 500 puzzles he wrote, he chose five of the hardest for the benchmark. In the example he walks through, called Hoofdrain, the winning line requires playing creatures, attacking and draining in the right order, and 10 of 20 models solved it; GPT 6 Luna failed by casting its key creature, Craterhoof, too early. The full results and cost per puzzle appear in images in the post. Sean names two limits: Forge offers only legal moves, so the benchmark cannot test whether a model knows the rules, and single-turn puzzles do not test planning over a longer game.