LangChain Benchmarks a Non-Generative Model as an Agent-Eval Judge
- Source
- x.com
- Date
LangChain built a test agent with its open source Deep Agents harness, captured five weather-task runs as a fixed LangSmith dataset, and scored each one with GPT-5.6 Luna, GPT-5.6 Terra, Claude Sonnet 4.6, and TypeSafe AI's Jev. Jev is not a generative model: it returns typed answers and probabilities directly instead of producing text a parser then has to interpret. Against a human reviewer's labels as the oracle, Jev matched every one of 500 repeated pass or fail decisions, while Terra matched 99.8 percent, Luna 96.4 percent, and Claude 80.0 percent. Jev's quality score variance also ran 92 to 913 times lower than the LLM judges, and it averaged 0.44 seconds per call at $0.00035, versus $28.17 total for Claude across the same runs. The authors call the result observational rather than causal, since the experiment is narrow and Jev's training objective, not just its architecture, could explain the gap. They also flag a real risk: a cheap, low variance judge that is consistently wrong can still produce bad feedback at scale, so teams still need human review in the loop. If it holds up on harder tasks, typed classifiers become a third category of agent evaluator, alongside code-based checks and LLM-as-judge.
We tested Jev against LLM judges on accuracy, repeatability, latency, and cost to see whether System One models could offer a new approach to agent evaluation. https://t.co/hrqdNpm0g8
Context
LangChain tested whether Jev, a model from TypeSafe AI that returns typed answers with probabilities instead of generating text, can serve as the judge in evaluations, and its write-up found that on a small test Jev matched the best language-model judge on pass/fail decisions while being far more consistent and far cheaper. Agent evaluations usually rely on code checks, which only work for narrow, fixed inputs, or on a that reads the agent's trace and scores it, which LangChain describes as slow, expensive and non-deterministic. TypeSafe calls Jev a System One model: it evaluates a state and returns structured decisions that software can use directly.
LangChain built a weather agent with its Deep Agents , captured its responses to five weather requests, and froze them in LangSmith so every judge saw identical material. A human reviewer labeled each response to serve as the reference. Jev, GPT-5.6 Luna, GPT-5.6 Terra and Claude Sonnet 4.6 each judged every case 100 times, giving a pass/fail decision and a continuous quality score. On pass/fail, Jev agreed with the human on all 500 decisions, Terra on 99.8%, Luna on 96.4% and Claude Sonnet 4.6 on 80.0%. The wider gap was in consistency: the variance of each judge's quality scores across repeats of the same case was 92 times Jev's for Claude, 433 times for Luna and 913 times for Terra. Jev averaged 0.44 seconds and $0.00035 per call, and the whole run cost $0.34 with Jev against $28.17 with Claude.
The 500 decisions are five cases judged 100 times each, so the accuracy result shows agreement with one reviewer on five outputs, not on 500 different agent behaviors. LangChain calls the results early, says the experiment cannot explain why Jev varied less, and warns that a cheap judge that is consistently wrong would spread bad feedback at scale. The language-model judges ran at their providers' default settings, with no temperature or seed fixed, and the Jev service version was not recorded. The test supports trying Jev where many repeated, cheap judgments matter, such as scoring production traces, while its accuracy on harder agent tasks remains untested here. OpenRouter later published its own comparison of Jev and an LLM judge.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
Checking sign-in…
Loading comments…