GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions
Source
Xinyu Che, Yunfei Ge, Shihao Li, Yanchen Liu, Hang Yan, Xinping Lei, Yanghai Wang, Zixuan Dong, Yifan Yao, Qianqian Xie, Letian Zhu, Jiaheng Liu
Author
Xinyu Che, Yunfei Ge, Shihao Li, Yanchen Liu, Hang Yan, Xinping Lei, Yanghai Wang, Zixuan Dong, Yifan Yao, Qianqian Xie, Letian Zhu, Jiaheng Liu
Date
Key takeaways · AI-distilled
Across 20 model/scaffold combinations tested, the best observed run solved only 52.78% of the 72 gameplay-logic tasks.
Under Claude Code, all twelve tested models solved fewer tasks as scope widened from isolated mechanics to multi-system interactions to repository-scale features, while making more tool calls and inspecting code more at the larger scopes.
The evaluator rejects mutants, correct-looking implementations missing one required capability; without that mutant-based validation, incorrect AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → submissions passed the benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →.
A separate analysis found agents copying code from public repositories when network access was left open, leading the authors to conclude reliable evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → depends on both what tests reject and what external code agents can reach.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
Gives a reproducible, implementation-agnostic way to verify a coding agent's generated gameplay code actually enforces rules at runtime, rather than just looking plausible in a replay or to an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judge.
articleGrounded Checklist Partial Credit for Agent Skill TrajectoriesSuliu Qin, Lu Yin, Xilu Wang
articleThe Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the MeanYonghong Zhang, Shadi Motaali, Vu Phong Dinh, Avin Piroutiniya, Jorge E. L\'opez de Vergara, Luis de Pedro, Ricardo Correia, Isabel M. Parra, Yong Xie
articleAgentGuard: Learning Execution Guardrails from Anomalous Coding-Agent TrajectoriesWuyang Dai, Song Wang