Finding the Move Is Not Winning the Game: XiangqiBench for Closed-Loop Evaluation of LLM Agents
- Source
- arxiv.org
- Author
- Yekun Chai, Qiwei Peng, Haoyi Xiong
- Date

Context
Picking the right move is not the same as winning, and a called XiangqiBench measures the difference for language-model agents. It uses 119 endgame positions in xiangqi (Chinese chess) where a forced checkmate exists. The must deliver mate against an engine opponent, using an interface that separates real moves, state queries and forward simulation. The authors recorded 8,568 multi-turn games from 12 frontier models.
They report three gaps. Under one of two observation protocols (called Sighted), models played the stored reference first move in 26.1% of trials, yet only 13.9% of those trials ended in a win. The leading model reached 38.7% pass@3 (a win in at least one of three tries) but 5.9% pass^3 (wins in all three). And 32.3% of accepted simulation calls stopped on an illegal move, while in 49.3% of comparable cases the real defender replied differently from the line the agent simulated.
The authors conclude that agent evaluations should score closed-loop outcomes and report reliability alongside coverage. The results come from one game and one set of positions, and only the abstract was available here.
- XiangqiBench gives an agent 119 tactical Chinese chess endgames with forced mates and requires it to actually deliver checkmate against an engine defender, recording 8,568 multi-turn trajectories from 12 frontier LLMs.
- Conversion gap: models played the stored correct first move in 26.1% of sighted trials, yet only 13.9% of those trials ended in a win, so naming the right move says little about finishing the job.
- Consistency gap: the leading model hit 38.7% pass@3 but only 5.9% pass^3, winning all three attempts on just 7 of the 46 positions it ever won, which is why the authors want reliability reported next to coverage.
- Simulation gap: 32.3% of accepted simulation calls stopped on an illegal move, and in 49.3% of comparable cases the real defender replied differently than the agent's simulated line, so self-written rollouts cannot anticipate the opponent.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleBacktrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQsRuoxi Zhao, Maziar Raissi
articleWhen Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-benchSungbin Im
articleAgentic Data Engineering for LLMs: System Design and a Controlled Empirical StudyRuomeng Ding, Qianli Shen, Zhaoyang Han, Meixin Chen, Daoyuan Chen, Yaliang Li
Checking sign-in…
Loading comments…