Intel / article

How To Use Decision Models for Evals

Source: hamel.dev
Date: Published Oct 20, 2026

Context

Decision models are a new class of classifier APIs that answer a narrow question, such as whether a trace contains a failure, with a probability instead of a written reply. They are being pitched as a cheaper substitute for judges in automated . Hamel Husain and Antaripa Saha argue that the number most teams leave untouched, the 0.50 cutoff that turns a probability into a pass or fail, should be tuned before any model is compared (hamel.dev). Their illustration: if a model scored every failure 0.40 and every passing trace 0.10, it would separate them perfectly at a cutoff of 0.30, yet the default would flag nothing. Moving the threshold trades precision (how many flagged traces really fail) against recall (how many real failures get caught). The authors set a minimum of 80% for each, then picked thresholds on training splits of their labeled data and measured the results on held-out traces.

They tested six decision models on 98 human-labeled traces from an apartment-leasing support , with a deliberately broad task: find any substantive failure. Three models could reach both minimums. OpenAI Decisions, which runs on GPT-6 Luna, scored best with an F1 of 0.87 at 3.72 cents per 100 evals. Perplexity's Decider was the cheapest qualifying option at 0.82 cents, with an F1 of 0.79. Liquid's d1-ultra also qualified. The post also checks whether scores can be read as real probabilities. In one group of Cloudflare Clef Flash traces, the average predicted failure chance was about 8%, but 60% actually failed. A simple correction fitted to labeled data, Platt scaling, helped some models and made OpenAI's slightly worse.

This is one small dataset and one broad task, and the authors say results vary a lot between tasks. They expect the gaps to narrow with a separate, narrower judge for each failure mode. Liquid launched d1-ultra as trained for better-calibrated probabilities, and here it needed little adjustment, which is consistent with that claim on this data but does not test it more broadly.

Key takeaways · AI-distilled

The default 0.50 threshold can miss everything: a model scoring every failure 0.40 and every pass 0.10 is perfect at 0.30 yet flags nothing at 0.50, so Hamel Husain's team tunes thresholds on labeled data.
Set minimums before chasing F1. For alerting, the author wants at least 80% precision and 80% recall; only Perplexity Decider, Liquid d1-ultra and OpenAI Decisions met that bar on the apartment-leasing traces.
After tuning, Perplexity was cheapest at 0.82 cents per 100 eval runs with F1 0.79, while OpenAI Decisions (GPT-6 Luna) reached F1 0.87 at 3.72 cents, about 4.5 times the cost.
Scores are not automatically probabilities: one Clef Flash group averaged about 8% predicted failure while 60% actually failed. Platt scaling helped Perplexity, d1 and Clef Flash but the Platt fit they used slightly worsened OpenAI's.
The authors recommend a separate judge per failure mode, tuning thresholds with stratified five-fold cross-validation on saved scores, and measuring yourself, since performance varied substantially across tasks and datasets.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.

Read the source hamel.dev