Intel / article

Tracing Agent Harness Behavior with NVIDIA NeMo Relay

Source: developer.nvidia.com
Author: William Markito Oliveira
Date: Published Sep 30, 2026

Context

An that makes fewer calls may simply have given up sooner. NVIDIA's NeMo Relay tutorial shows how to tell that apart from efficiency by pairing a task-success check with a trace of model calls, tool calls, errors and timing. The trace records the route to the answer, including recovery attempts that a final score hides. The tutorial demonstrates this with Hermes Agent, an that integrates NeMo Relay.

Its case study compared a baseline Hermes with a batch of tool-layer fixes across nine tasks, each run three times per model. For Qwen3 Coder 30B, the fixes raised completions from 19 of 27 runs to 22 of 27, while mean duration rose from 27 to 42 seconds and mean tool calls from 2.8 to 3.9. The authors attribute the extra turns on one task to recovering where the baseline had stopped, and they traced a regression on another task to a specific output. Those measurements let developers weigh additional completions against longer waits instead of treating fewer calls as an improvement by itself. Case-study results.

Key takeaways · AI-distilled

An ATIF trajectory shows which tool the model asked to run, not whether it worked. The authors say to confirm outcomes in the ATOF stream, where matching start and end events share a uuid and parent_uuid links a tool call to its parent.
NVIDIA's recipe for judging a harness change: alter one thing, hold model snapshot, provider, input, budget and timeout fixed, run equal repetitions per arm, and compare verified task outcomes before looking at calls, or cost.
In the 108-run ToolPerf rerun, Claude Sonnet 4.5 was effectively unchanged (24/27 vs 23/27). Qwen3 Coder 30B rose from 19/27 to 22/27, but mean calls went from 3.8 to 4.9, tool data from 16 KB to 33 KB and duration from 27 s to 42 s.
The task audit shows the trade-off: recovery recipes lifted Qwen on a blocked-command task from 33% to 100%, while zero-match probe output on a case-insensitive search pushed turns from 3.3 to 9.3, a regression the authors flag.
The authors caution that runs using live web search are for exploring behavior, not ranking models; a fair comparison needs fixed search responses and repeated runs. Traces can contain prompts, tool arguments and file paths, so review them before sharing.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

Read the source developer.nvidia.com