Intel / article

Use NVIDIA NeMo to Turn Agent Failures into Tests and Harness Fixes

Source: developer.nvidia.com
Author: Patrick Moorhead
Date: Published Nov 4, 2026

Context

NVIDIA has published a tutorial for improving an by repairing its , the instructions, tool definitions, routing and other code around the model, rather than the model itself. NeMo Compass reads stored records of agent runs, called traces, and turns them into a list of recurring problems, each linked to the trace IDs that show it. NeMo Eval Author gives a coding agent such as Codex or Claude Code the skills to write evaluations that reproduce those problems. The accompanying blueprint also includes NeMo Relay, NVIDIA's agent tracing library, though the post does not describe its part in this loop. A developer then changes one part of the harness and reruns the evaluations to see whether the change helped.

The worked example is a Hermes Agent assistant for a fictional company, connected to its chat, email, files and tickets. Asked to "draft a message with project updates," it drafted the message but also called its send-message tool without asking for confirmation. Compass, run on 42 recorded traces and a short file describing the assistant's expected behavior, flagged the unconfirmed sends by comparing the agent's actions with those rules. The coding agent then wrote four evaluations with a check that fails any run in which the agent sends without confirmation. The fix was a short passage in the agent's SOUL.md, Hermes's equivalent of a , stating that a request to draft a message authorizes preparation only, and that a token returned by a preparation tool "does not establish the user's consent" to send.

On two development tasks run three times each, passes went from 3 of 6 to 6 of 6 and tool calls from 10 to 6. On two held-out tasks, kept hidden from the coding agents proposing changes to test whether the fix generalizes, passes went from 4 of 6 to 6 of 6 and tool calls from 28 to 15. A second Compass pass no longer reported the sending problem and found a new one: some answers were not in the evidence the agent had retrieved. NVIDIA calls this a small set of tasks focused narrowly on the Compass findings and says production suites often contain hundreds to a few thousand examples. An earlier NeMo Relay tutorial shows why the traces matter alongside pass rates: for one model, its harness fixes produced more completed tasks but also longer runs and more tool calls.

The post also applies the loop to LEO, Dassault Systèmes' engineering assistant, on a car door design task. Compass found that the agent repeatedly put arguments to a model-query tool outside the expected field, and the tool returned an empty error message the agent could not recover from. A coding agent added a validation step for misplaced arguments, and malformed tool calls fell from 142 to zero with accuracy unchanged. In an overnight run of about 12 hours, the post reports a 28% relative improvement in the workflow's overall evaluation score and 35% fewer tool calls per run across six scenarios, with no regressions reported elsewhere. These figures are as reported in NVIDIA's post, without independent measurement. The post does not define the overall score or give its starting value, and its claim that Compass found anomalies a coding agent missed in internal benchmarks comes with no numbers.

Key takeaways · AI-distilled

The loop starts from stored traces: NeMo Compass read 42 JSONL Hermes traces plus ETHOS.md, the file defining expected behavior, and flagged runs where the agent called send_message without user confirmation, linking each signal to trace IDs.
The fix was a prompt rule, not a model change. Hermes's SOUL.md was edited to say a request to draft authorizes preparation only, and that a token returned by a preparation tool is a technical capability, not the user's consent to act.
Two of the four generated eval tasks were withheld from the coding agent proposing harness changes, and NVIDIA's rule is to accept a change only when it improves both sets, keeping held-out cases hidden to avoid overfitting.
In the Dassault Systemes car-door example the agent was stuck on an empty error message it could not recover from, caused by QueryModel arguments placed outside the expected options field; a validation step cleared it with accuracy unchanged.
NVIDIA says production eval suites often contain hundreds to a few thousand examples, and the engineering example's optimization loop ran unattended overnight, roughly 12 hours, before it reported its gains.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.

Read the source developer.nvidia.com