Build your own decision model

For fixed-choice classification, masking logits to the allowed answers takes one forward pass instead of one per generated token. The post gives working code and cautions that token probabilities are not calibrated confidence.

• token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
• structured output — Forcing a model's response to match a schema, so downstream code can parse it instead of guessing at prose.
• calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.

• articleHow Cua Replaced Full LLM Calls with Fast Decision ModelsCua
• articleGeneration configurations: temperature, top-k, top-p, and test time computeChip Huyen
• articleJev introduces a new shape of LLM - System One, aka Decision Modelssimonwillison.net

Checking sign-in…

Loading comments…