Intel / article

How to Evaluate AI Agents From Tool Calls to Task Completion

Source: developer.nvidia.com
Author: Elizabeth Goodman
Date: Published Sep 21, 2026

Context

An 's successful tool call does not by itself show that the user's task was completed. NVIDIA's evaluation guide separates checking individual actions from checking the resulting state of the system. In its refund example, a correctly formed request is only one part of establishing that the refund actually posted.

The guide proposes keeping an ordered record of actions to diagnose where a task failed, while judging completion against the intended outcome. Repeating the evaluation shows how reliably that outcome occurs; measuring cost per successful task includes the burden of failed attempts. This gives a team both an operational result to judge and enough detail to investigate what needs fixing.

Key takeaways · AI-distilled

Metrics roll up through a fixed hierarchy: Benchmark -> Trial -> Task -> Turn -> Step, where a step is one atomic tool call or non-tool emission such as a plan or the final message.
Report accuracy paired with its consistency range across 3-5 trials, not a single number: a model hitting 90% then 74% is a worse bet than one holding steady at 84%, even though both average 82%.
-as-a-Judge scores should be treated as provisional until validated against human ratings on a sample; executable checks (did the database update, did tests pass) are the preferred gold standard.
Nemotron 3.5 Lightning's 86% PinchBench accuracy comes with finishing 10,000 tasks 30% faster than Qwen3.6 35B at comparable accuracy, so the comparison is on speed as well as correctness.
A worked trace shows the metrics diverging: the task passed (E2E score 1) even though one step, viewing a 400+ line file instead of grepping for the class, was scored redundant, pulling step-level score and tool-call precision to 3/4.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.

Read the source developer.nvidia.com