An Independent Evaluation of TypeSafe's Jev

Jev, a model from TypeSafe, does not write text. It takes a document or other data plus a set of questions, each with a fixed list of allowed answers, and returns a probability for every option. Because the usual benchmarks run by Vals grade written answers, Vals built two tests Jev could take for its independent evaluation: 400 claim-verification items made from SEC 10-K filings, and a preregistered slice of LegalBench, a public set of legal-reasoning tasks, with 396 questions across 12 subtasks. The result splits cleanly: Jev holds up on bounded checking against a document and falls behind on the legal questions.

In claim verification, a system reads a claim and an excerpt and decides whether the excerpt supports it, contradicts it, or does not address it. The correct answers came from how each item was constructed, not from a model, and a human audited every item. All twelve systems scored between 0.953 and 0.995, and Jev's 0.975 tied GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna. The gap was in price and speed. Jev cost about 2 cents per 1,000 cases, roughly 1/498th of Astra's cost, and answered in about a tenth of a second whether it was asked for one judgment about a document or thirty-two; at thirty-two, its median latency was 194 times lower than Astra's. Cheap LLMs narrow that advantage without erasing it: GPT-6 Luna matched Jev's score at about three times its price.

LegalBench went the other way. Jev finished last of twelve on both raw and class-balanced accuracy, scoring 0.730 on the balanced measure against 0.932 for the leader, GPT-5.6 Terra. It did worst on the two contract-entailment subtasks, the closest match to claim verification, answering yes to all 33 items in one of them. Its calibration, meaning how well its stated confidence matched its actual accuracy, was the best of the twelve on claim verification and the worst on LegalBench, so a confidence level measured on one task did not hold on the other. Vals cautions that each subtask has only 33 items, that the slice does not represent the full benchmark, and that LegalBench is public and probably in the LLMs' training data.

Vals also checked TypeSafe's own numbers. The launch post claimed Jev was 444.6 times cheaper and 193.6 times faster than the average of GPT-6 Astra and Claude Fable 5.1. Vals found the cost figure holds against Astra, and its 194-times speed gap at thirty-two judgments matches the latency claim, but on single judgments under load the gap shrank to 47 times. When each system's confidence threshold was set on 30% of items to stay within a 1% error budget and then applied to the rest, Jev handled 95% of cases at 1.6% error, one of six systems that went over the budget.