TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents
Source
arxiv.org
Author
Krishanu Adhikary, Prashant Devadiga
Date
Why it matters
Grading tool-using agents only on final records lets them skip required steps and still pass. Checking every required action and unrequested changes gives a far lower, more honest success rate.
Key takeaways · AI-distilled
TauIndianBankBench has 1,000 Indian retail-banking tasks in which an assistant helps a simulated customer, under bank rules that prescribe required steps, such as obtaining a closure quote before closing a deposit.
Many required steps leave the account records unchanged, so grading only the final records cannot see them. The benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → also checks that the assistant took every required step and made no unrequested change.
For every model tested, 6.5-12.0% of sessions are judged successful under a records-only check even though the assistant skipped a required step or made an unrequested change.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleBacktrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQsRuoxi Zhao, Maziar Raissi
articleFinSkillBench: Evaluating AI Agents and Domain Skills for Investment ManagementJermyn Zhen Yong Bek, Zhuang Qiang Bok, Zhongtian Sun
articleToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling AgentsYiShan Zheng, Yuan Wu, Yi Chang