When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline
Source
Stefan G. Creadore, Peyton Woakz
Author
Stefan G. Creadore, Peyton Woakz
Date
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
It documents concrete ways AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → numbers get distorted: passing gates that exempt known gaps, 32-bit-clamped durations inflating latency percentiles, and a 94% reduction figure that is really 46% once the full call chain counts.
articleEvaluating Enterprise Analytics Agents: An End-to-End, Trace-Backed MethodologyTeja Venkat Kolli, Sang Su Lee, Xueying Yan, Jessie Chen, Chi Cheng, Kartik Ravisankar, Shishir Dash, Vijay Anand Raghavan
articleFrom Traceability to Justifiability: Accountability Structures in Agentic Software EngineeringRashid Azarang
articleThe Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the MeanYonghong Zhang, Shadi Motaali, Vu Phong Dinh, Avin Piroutiniya, Jorge E. L\'opez de Vergara, Luis de Pedro, Ricardo Correia, Isabel M. Parra, Yong Xie