When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline
Source
arxiv.org
Author
Stefan G. Creadore, Peyton Woakz
Date
Context
A release check can report no blocking failures while many evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → cases still fail. An audit of the Praxa AI pipeline found 112 passing cases and 27 failures in a historical 139-case routing report. All failures were classified as known gaps exempt from blocking the development gate.
Those figures describe different things: the gate records whether policy permits progress, while the case pass rate records success on the selected tests. Treating a clean gate as perfect performance would erase the exemptions. The audit reconstructs historical records; it does not establish current deployment reliability or a general capability ranking.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
articleEvaluating Enterprise Analytics Agents: An End-to-End, Trace-Backed MethodologyTeja Venkat Kolli, Sang Su Lee, Xueying Yan, Jessie Chen, Chi Cheng, Kartik Ravisankar, Shishir Dash, Vijay Anand Raghavan
articleHow Do Coding Agents Optimize Software and Report Performance Validation? A Large-Scale Empirical Study of Open-Source Pull RequestsHuiyun Peng, Ricardo Calvo, Kelechi G. Kalu, James C. Davis
articleNot All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the WildObada Kraishan