Intel / article

Create a LangChain Deep Agents Harness Profile for NVIDIA Nemotron 3 Ultra to Improve Performance

Source: developer.nvidia.com
Author: Sean Lopp
Date: Published Jul 8, 2026

Context

An can answer incorrectly because its tool interface hides what it has not read. In NVIDIA's Nemotron 3 Ultra example using LangChain Deep Agents, a file-reading tool returned only the first page, but the model answered a question about the file's final line. The repair added a notice explaining that a full page probably meant more content remained and how to request it. In the reported evaluation, the targeted check changed from zero passes in three runs to three of three, and the broader average rose from 94 to 96 successes out of 127. The change touched only the surrounding software, not the model's training, and because the benchmark is stochastic, the two-task gain is modest.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.

Key quotes

“The goal of harness engineering is to make the calls from the agent to the model more closely resemble what the model saw in the training data.”
Sean Lopp

“Geoffrey Huntley calls this the “ralph loop”: a model repeatedly reads the current state, makes one change, and uses the filesystem—not the conversation—as memory.”
Sean Lopp

“A fix is accepted only after it passes the same test several times in a row, which filters out changes that passed once by luck.”
Sean Lopp

“And it is told to prefer the general fix over the specific hack—to infer the broader behavior a failure exposes rather than special-casing the exact inputs of one test. This distinction is the difference between teaching the model a durable skill and memorizing a single answer.”
Sean Lopp

“The evaluation benchmark is what makes this safe to automate—it gives the loop a ground-truth verifier, so a proposed change is only kept if it measurably helps.”
Sean Lopp

Read the source developer.nvidia.com