LangChain Benchmarks a Non-Generative Model as an Agent-Eval Judge
Source
LangChain
Author
LangChain
Date
From the Daily Brief
LangChain built a test agent with its open source Deep Agents harness, captured five weather-task runs as a fixed LangSmith dataset, and scored each one with GPT-5.6 Luna, GPT-5.6 Terra, Claude Sonnet 4.6, and TypeSafe AI's Jev. Jev is not a generative model: it returns typed answers and probabilities directly instead of producing text a parser then has to interpret. Against a human reviewer's labels as the oracle, Jev matched every one of 500 repeated pass or fail decisions, while Terra matched 99.8 percent, Luna 96.4 percent, and Claude 80.0 percent. Jev's quality score variance also ran 92 to 913 times lower than the LLM judges, and it averaged 0.44 seconds per call at $0.00035, versus $28.17 total for Claude across the same runs. The authors call the result observational rather than causal, since the experiment is narrow and Jev's training objective, not just its architecture, could explain the gap. They also flag a real risk: a cheap, low variance judge that is consistently wrong can still produce bad feedback at scale, so teams still need human review in the loop. If it holds up on harder tasks, typed classifiers become a third category of agent evaluator, alongside code-based checks and LLM-as-judge.
Jev matched the human-labeled oracle on all 500 repeated binary pass/fail decisions across the test set; GPT-5.6 Terra matched 99.8%, Luna 96.4%, and Claude Sonnet 4.6 only 80.0%.
Jev averaged 0.44 seconds per call in the experiment; TypeSafe AI separately claims up to 200x faster inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → and 400x lower cost than comparable LLMs on general classification tasks.
The test built a target weather AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → with the open-source Deep Agents agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition →, captured five fixed runs as a LangSmith dataset, and used a human reviewer's labels as the accuracy oracle for every judge.
Jev supports three typed question forms evaluated in parallel against the same state: Choice (pick one option with confidence), Score (rate against a rubric), and Noul (probability a yes/no judgment is true).
The authors call the lower-variance result observational rather than causal, and warn that a cheap per-call judge can also amplify a consistently wrong evaluator's mistakes at scale if teams skip human review.
Terms in this piece · Glossary
LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
