Intel / article

OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment

Source: arxiv.org
Author: Seonglae Cho, Adriano Koshiyama
Date: Published Jul 30, 2026

Context

OptimismBench tests whether a 's probability estimates for an outcome and for its opposite are consistent. For example, assigning 70% to a startup's success and 15% to its failure leaves 15 points unaccounted for. The benchmark asks for both framings of each scenario and uses the asymmetry as a signed bias score, without needing to know the event's true probability. The authors report an optimistic direction in 14 of 16 tested models. This gives users of AI forecasts a way to audit a model's directional tilt, although consistency between the two answers would still not establish that either forecast is accurate. Research abstract.

Key takeaways · AI-distilled

The trick is inverted pairs: ask one model for P(success) and P(failure) on the same scenario. They should sum to 1, and the gap is a signed bias score needing no ground truth — which is exactly what metrics miss, since they average unsigned error.
Across 16 models from 8 providers, 14 tilt optimistic. Pessimism showed up only in Anthropic's frontier tier.
Post-training, not , sets the direction of the tilt. Eleven matched base-versus-chat pairs across four model families shifted sign after , and different families shifted opposite ways.
The tilt belongs to the model, not the wording. It survived changes to prompt, temperature, perspective, and explicit self-debiasing, and across six languages model identity mattered 4.7x more than the language used.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.

Key quotes

“When an LLM rates a startup's success at 70% but its failure at 15%, the missing 15 points expose a distortion no aggregate score flags.”

“Across 16 models from 8 providers, fourteen are optimistic; pessimism appears only in Anthropic's frontier tier.”

“Eleven matched base-versus-chat pairs across four families show post-training sets the sign of the bias, with opposite shifts in different families.”

“When alignment makes a model more helpful, it also tilts its probabilities; downstream pipelines inherit the tilt by default.”

Read the source arxiv.org