Vibeleaderboard
← All Intel
Intel / article

Finding the Move Is Not Winning the Game: XiangqiBench for Closed-Loop Evaluation of LLM Agents

Source
arxiv.org
Author
Yekun Chai, Qiwei Peng, Haoyi Xiong
Date

Context

Picking the right move is not the same as winning, and a called XiangqiBench measures the difference for language-model agents. It uses 119 endgame positions in xiangqi (Chinese chess) where a forced checkmate exists. The must deliver mate against an engine opponent, using an interface that separates real moves, state queries and forward simulation. The authors recorded 8,568 multi-turn games from 12 frontier models.

They report three gaps. Under one of two observation protocols (called Sighted), models played the stored reference first move in 26.1% of trials, yet only 13.9% of those trials ended in a win. The leading model reached 38.7% pass@3 (a win in at least one of three tries) but 5.9% pass^3 (wins in all three). And 32.3% of accepted simulation calls stopped on an illegal move, while in 49.3% of comparable cases the real defender replied differently from the line the agent simulated.

The authors conclude that agent evaluations should score closed-loop outcomes and report reliability alongside coverage. The results come from one game and one set of positions, and only the abstract was available here.

Key takeaways · AI-distilled
  • XiangqiBench gives an agent 119 tactical Chinese chess endgames with forced mates and requires it to actually deliver checkmate against an engine defender, recording 8,568 multi-turn trajectories from 12 frontier LLMs.
  • Conversion gap: models played the stored correct first move in 26.1% of sighted trials, yet only 13.9% of those trials ended in a win, so naming the right move says little about finishing the job.
  • Consistency gap: the leading model hit 38.7% pass@3 but only 5.9% pass^3, winning all three attempts on just 7 of the 46 positions it ever won, which is why the authors want reliability reported next to coverage.
  • Simulation gap: 32.3% of accepted simulation calls stopped on an illegal move, and in 49.3% of comparable cases the real defender replied differently than the agent's simulated line, so self-written rollouts cannot anticipate the opponent.
Terms in this piece · Glossary
  • benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Recommended reads
Comments

Checking sign-in…

Loading comments…