Evaluating Enterprise Analytics Agents: An End-to-End, Trace-Backed Methodology
Source
Teja Venkat Kolli, Sang Su Lee, Xueying Yan, Jessie Chen, Chi Cheng, Kartik Ravisankar, Shishir Dash, Vijay Anand Raghavan
Author
Teja Venkat Kolli, Sang Su Lee, Xueying Yan, Jessie Chen, Chi Cheng, Kartik Ravisankar, Shishir Dash, Vijay Anand Raghavan
Date
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
Grading only final answers hides where analytics agents actually fail, like using the wrong source of truth or skipping a required decomposition; this methodology gives builders a way to catch those failures before they reach a business decision.
Key quotes
“Enterprise analytics agents are not only text-to-SQL systems.”
“Grading final answers hides where these agents fail.”
“These results show why trust in analytics agents requires end-to-end, trace-backed evaluation rather than SQL correctness or final answer quality alone.”
