A practical framework for evaluating agentic AI systems

Kamil Pluciński, a senior machine learning engineer at the AI consultancy deepsense.ai, argues that a passing evaluation score is weak evidence that an AI agent is ready to run a real business process, and sets out what stronger evidence looks like (deepsense.ai). The difference from evaluating an ordinary model is that an agent takes actions, such as issuing a refund or updating a ticket, and its result depends on more than the model: its tools, their permissions, prompts, retries and the state of the surrounding systems can all change what it does. The post follows an earlier deepsense.ai article on choosing among several candidate systems; this one asks how to build evidence before an agent has any production traffic to learn from.

His central rule concerns repetition. The same agent with the same input, tools and starting state can take a different path on each run, even with temperature set to 0, so one run is only one sample. He recommends running every test case several times and gating releases on pass^k, which requires all k runs to succeed, instead of pass@k, which counts a case as solved if any single run succeeds and so shows what the agent can do on a good day. He applies the same thinking to the route the agent takes. An agent can reach the right final state after calling the wrong tool, retrying blindly or wandering through a detour, so each case should carry several narrow checks on tool choice, call order, handling of failed tool calls, and cost and latency, while leaving the rest of the path free.

Scores from a language model acting as judge get the same scrutiny. Pluciński says such judges usually cannot hold a 1-to-10 scale in a consistent order, so he favors yes-or-no checks, such as whether the refund was issued, and wants every judge measured against human labels on a held-out set, using Cohen's kappa, a statistic for agreement between raters. His worked example is deepsense.ai's own account of a technical-support agent in a supply-chain operation: production failures were turned into test cases until the set reached about a thousand, and when the system was later restructured, that set came back at the same level and replaced the weeks of manual verification such a refactor usually needs. He is explicit that none of this makes an agent safe to deploy; it lets a team know what it checked and what it did not.