Jev Vs Llm As A Judge

Teams commonly grade an AI agent's outputs by asking a second LLM to write a verdict, a pattern called LLM-as-a-judge, which an earlier OpenRouter tutorial covers. This OpenRouter post asks whether TypeSafe's decision model Jev can do that grading instead (openrouter.ai). Its central point is about the confidence number. When an LLM judge reports a confidence, that number is text the model chose to write. Jev returns a probability meant to match how often answers like this one turn out acceptable, which can be checked against labels and used to set a threshold.

On 88 question-and-answer items drawn from a public dataset and reviewed before scoring, where the judge had to decide whether an answer stuck to a supplied passage, accuracy was a tie: Jev matched the labels on 84 and the LLM judge, GPT Luna, on 83. The difference was calibration. Jev's Brier score, the average squared gap between stated probability and the true outcome, was 0.043 against 0.054. The LLM judge used only six distinct confidence values, so every case sat near 0 or near 1, and cases it scored near 0 were actually acceptable 10 percent of the time. Jev put seven cases between 0.3 and 0.7, the ones worth routing to a person. Jev cost $0.021 per 1,000 judgments at a 171 millisecond median, against $0.114 and 1,662 milliseconds for the LLM judge, which the authors say matters most when the judge sits in a live request path.

The LLM judge won where the task was more open. On 50 news summaries rated by experts, the two tied on coherence, but on factual consistency the LLM judge's rankings tracked the experts much more closely, a correlation of 0.72 against Jev's 0.47. The authors guess this is because checking consistency means working claim by claim through a long article, which suits a generative model. Both judges still caught the three summaries experts rated low on consistency. The authors' split: Jev for closed rubrics where the evidence is in the request, an LLM judge for open rubrics, long texts, or when someone needs a written reason, plain code for anything it can check exactly, and a person for consequential, novel calls. The datasets are small and were run on a single day, September 21, so the post presents its numbers as the shape of the trade-off and tells teams to rebuild the comparison on their own labeled data.