Grounded Checklist Partial Credit for Agent Skill Trajectories
Source
arxiv.org
Author
Suliu Qin, Lu Yin, Xilu Wang
Date
Context
Two AI-AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → runs can both fail while completing different amounts of useful work. Grounded Checklist Partial Credit measures that difference with task-specific requirements generated under human-defined rules. A judge scores each requirement using execution-log evidence and abstains when the log cannot establish an answer.
Among 879 matched pairs that failed both with and without an added procedural agent skillA reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.Full definition →, the authors found that 20.9% improved their checklist score by more than 0.10, while 18.7% worsened by that margin. A pass-or-fail summary would miss both changes. The method gives developers a way to locate partial improvements and regressions while keeping the official task outcome as a separate result.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleInducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent EvaluationDarragh Quinn, David Dylan, Roisin Healy, Fionn Carroll, Maeve Donnelly, Cormac Sheehan
articleGAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented AgentsUmesh Bodhwani, Thanh Tran, Kai Wei
articletrajectory-judge: What Outcome-Only LLM Judges Miss on Agent TrajectoriesHadi Mohammadi