Evaluating Enterprise Analytics Agents: An End-to-End, Trace-Backed Methodology
- Source
- arxiv.org
- Author
- Teja Venkat Kolli, Sang Su Lee, Xueying Yan, Jessie Chen, Chi Cheng, Kartik Ravisankar, Shishir Dash, Vijay Anand Raghavan
- Date
Context
An analytics can answer more business questions while remaining inconsistent about the data behind its answers. In a study of an internal marketplace agent, the configuration described as more capable produced answers using real data on 73% of runs, compared with 21% for the other configuration. Yet it changed its interpretation of tables across repeated runs for 41 of 50 questions. Answering more often was therefore a different achievement from interpreting the data consistently. The researchers report these findings from 300 runs.
Their follows the agent's recorded actions as well as its final response, checking how it interprets the question, executes the work and behaves across repetitions. That makes failures such as choosing the wrong data source visible even when the final explanation sounds plausible.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
“Enterprise analytics agents are not only text-to-SQL systems.”
“Grading final answers hides where these agents fail.”
“These results show why trust in analytics agents requires end-to-end, trace-backed evaluation rather than SQL correctness or final answer quality alone.”
Checking sign-in…
Loading comments…