An Evaluation of Inception's Mercury Decide

Context

Mercury Decide, a model from Inception, is not a chatbot. You give it a state and a list of allowed answers, and in one call it returns a single answer with a probability for every option. That is the same design as Jev from TypeSafe, and Vals AI ran Mercury Decide through the exact datasets, prompts and scoring it used in its earlier Jev evaluation, so its new write-up works as a direct comparison. Inception prices Mercury Decide at half Jev's rate, $0.021 per million input tokens, with output free.

On 400 claim-verification items the two cannot be told apart (0.978 against 0.975), but Mercury Decide costs about a quarter as much, $0.006 against $0.025 per 1,000 claims, the lowest of the thirteen systems Vals tested. On LegalBench it leads Jev 0.869 to 0.740 in accuracy, with most of the gap coming from two contract entailment subtasks where it scored 0.970 and Jev scored 0.485 and 0.576; Vals still says it falls behind frontier systems on LegalBench. Because these models return probabilities, a user can set a confidence threshold and send only uncertain cases elsewhere. Vals tuned such a threshold on 30% of the claims to meet a 1% error budget, froze it, and applied it to the rest: Mercury Decide automated 94% of the held-out claims at 0.8% error, within budget, while Jev automated 95% at 1.6%, over it. Sonnet 5.5, Opus 5.5 and Gemini 4 Argon automated about 99 to 100% within budget, but every tested was slower and more expensive.

The trade-off appears when many questions are asked about one long document in one call. Jev's latency stays near 0.1 seconds, while Mercury Decide's median rises from 0.16 seconds for one judgment to 0.40 seconds for 64, and its billed input tokens grow from about 1,700 to about 106,000, so a 64-judgment call costs $0.0022 against $0.00016 for Jev. Vals concludes that for claim verification over a document Mercury Decide gives the same accuracy for a quarter of Jev's price, while Jev scales better for many questions per call. Mercury Decide has no version pin, and Vals describes it as of October 8, 2026; Inception announced an update with a larger and claimed gains around the same time, so later results may differ.

Key takeaways · AI-distilled

Mercury Decide works like Jev, returning one answer with a probability for every allowed option in a single call, priced at $0.021 per million input tokens with free output, half Jev's rate.
Most of its LegalBench lead over Jev comes from two contract entailment subtasks, where it scored 0.970 on both against Jev's 0.485 and 0.576. Its weakest subtask was international citizenship questions at 0.70.
By claim category it beat Jev on subtle contradictions (0.988 vs 0.912) but trailed on direct support (0.963 vs 1.000). matched Jev on claims (ECE 0.011) and was better on LegalBench (0.058 vs 0.108).
Its latency grows with question count, from a 0.16s median for one judgment to 0.40s for 64, while Jev stays near 0.1s. Billed input rose from about 1,700 to 106,000 tokens, so many questions per document scale better on Jev.
Seven of thirteen systems stayed within the 1% error budget on held-out claims. Mercury Decide's 94% coverage trails Sonnet 5.5, Opus 5.5 and Gemini 4 Argon, which take about 99 to 100%.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.