For closed rubrics with evidence in the request, a probability-returning judge matched an LLM judge at a fifth of the cost and a tenth of the latency. Open-ended, long-context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → checks still need an LLM judge.
Key takeaways · AI-distilled
OpenRouter notes that an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judge's confidence score is just text the model chose to write, while Jev returns a probability per answer, which is a frequency claim that can be checked against outcomes.
In the post's faithfulness example, the LLM judge marked an answer unacceptable with confidence 1, while Jev returned 0.01, meaning roughly 1% of similar answers should turn out acceptable.
Jev questions come in three shapes: noul for yes/no pass/fail gates, choice for picking one of N labeled options such as pairwise preference, and score for ordered scales like coherence 1 to 5.
The post advises using neither judge when code can check a criterion exactly, and a person when a decision is consequential and novel. An LLM judge is still the pick when someone needs to read why an answer failed.
A calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition → probability lets you set a threshold and route the uncertain middle band to a human reviewer; if calibration does not hold, the author says the number is just decoration.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
