Container-Based Regression Evals for Managed Deep Agents

LangChain poses the problem in its announcement: after you add skills to an agent, swap its model or reword a tool description, how do you know you didn't break something that already worked? Its answer is that Managed Deep Agents now ships with Harbor, an evaluation framework, so tests run in a fresh container each time and the results are tracked in LangSmith, LangChain's tracing and evaluation service. In the accompanying video, Nathan from LangChain's product team separates two uses of evals: catching regressions, meaning checking that a change didn't lower performance, and hill climbing, meaning pushing a score upward against set criteria. The walkthrough is about the first.

Each eval has three parts, as he describes it: an environment, which is the files, data and starting state the agent begins from; a job, which is the instruction given to the agent in a markdown file; and a check, a verifier that judges whether the agent did the job. Harbor builds a container image around the agent and starts a clean one for every run. The video gives the reason: agents may touch the file system or run commands, and a fresh container keeps one test from polluting the next. On disk, each eval holds the instruction, a Harbor config file, a Dockerfile for the container and the tests, and he says most of the setup is generated so developers mostly edit the instructions and tests.

The demo uses a made-up research assistant that answers questions from an internal document set through a doc-search tool. A scaffold command can hand the work of writing evals to a coding agent; in the video it goes to Claude Code, which uses an eval-writing skill and pauses so a person can approve a human-readable task file for each eval. Harbor then runs a test script that writes a reward, a score Harbor reads to decide success. In the simpler hand-written example the script runs pytest checks: whether the agent cited only documents that exist, whether it admitted when the documents didn't cover the question, and whether it listed its sources. The runs appeared in LangSmith, with a reward of one on all three generated evals.

The video frames these suites as stability tests, not a way to measure how good an agent is. LangChain suggests keeping them current as the agent grows and running them nightly in CI as well as during development, so that trying a different model, tool or tool description shows up as a measurable change in LangSmith. Harbor's developers have separately described it as a general framework for agent evaluation and RL environments, which is the tool LangChain has built into its hosted product here.