ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents
Source
arxiv.org
Author
YiShan Zheng, Yuan Wu, Yi Chang
Date
Why it matters
Clean benchmarks overstate tool-calling reliability: models degrade sharply and unevenly under interface, intent, output and runtime perturbations, and the stage attribution tells you which part of your agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → to harden.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
articleHarness Engineering in LLM Tool Use via Agent-Native Reusable Tool PrimitivesHaibo Jin, Suijin Wang, Xucheng Yu, Haojing Luo, Haohan Wang
articleThe Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent BehaviorXiangzhe Xu, Hamidreza Saghir, Qianhui Wu, Marc-Alexandre C\^ot\'e, Tong Wang, Kiran Lakkaraju, Kexin Pei, Xiangyu Zhang
articleCheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?Hang He, Li Wang, Hao Chen, Yuchen Shao, Yuling Shi, Lisheng Wang, Peiyang Liu, Goose Lin, Zaiyuan Wang, Haiying Sun, Ting Su, Chengcheng Wan