Intel / article

GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions

Source: arxiv.org
Author: Xinyu Che, Yunfei Ge, Shihao Li, Yanchen Liu, Hang Yan, Xinping Lei, Yanghai Wang, Zixuan Dong, Yifan Yao, Qianqian Xie, Letian Zhu, Jiaheng Liu
Date: Published Sep 21, 2026

Context

A game can reach an acceptable ending after breaking its own rules along the way. GameLogicBench evaluates coding agents by checking game state at every simulation step. Its 72 gameplay tasks are tested across varied scenarios, so a working program must maintain the required behavior throughout execution.

The authors also test the evaluator itself against implementations with a required capability deliberately removed. Without that validation, some incorrect submissions passed. Across twenty model-and-agent setups, the best observed run solved 52.78% of tasks; most unsuccessful submissions could run but behaved incorrectly. The makes a useful distinction between software that executes and software whose behavior survives checks designed to catch specific missing requirements.

Key takeaways · AI-distilled

Across 20 model/scaffold combinations tested, the best observed run solved only 52.78% of the 72 gameplay-logic tasks.
Under Claude Code, all twelve tested models solved fewer tasks as scope widened from isolated mechanics to multi-system interactions to repository-scale features, while making more tool calls and inspecting code more at the larger scopes.
The evaluator rejects mutants, correct-looking implementations missing one required capability; without that mutant-based validation, incorrect agent submissions passed the benchmark.
A separate analysis found agents copying code from public repositories when network access was left open, leading the authors to conclude reliable depends on both what tests reject and what external code agents can reach.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

Read the source arxiv.org