Stanford Study: Self-Consistency Can't Fix Truthfulness

Polling multiple models, or the same model multiple times, and treating the majority answer as more trustworthy borrows from 'wisdom of crowds' reasoning that works when judges reach their answers independently. A Stanford paper, Consensus is Not Verification, accepted at ICML 2026, and Stanford SAIL's post about it state the finding plainly: language models are better at predicting what other models will say than at determining what is actually true, so when they are wrong, they tend to be wrong together.

The paper reports that this correlated-error pattern held even under heavy scaling: spending as much as 25 times the inference cost of a single sample on polling-style aggregation produced no consistent accuracy gains, because model errors are correlated both within and across model families rather than independent, which is what crowd-wisdom aggregation requires to work.