A chatbot can favor a political position by making a stronger case for it than for its opposite. Anthropic's paired-prompt evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → tests that behavior by asking about opposing positions on the same topic, then grading the responses for comparable engagement, analytical depth and supporting evidence. Claude Sonnet 4.5 received a 94% even-handedness score across 1,350 prompt pairs. The November 2025 study used an automated model grader and mostly US political topics in single-response exchanges. Its result describes balance under that rubric; other dimensions of political bias require different tests.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
articleElection Safeguards UpdateAnthropic News
articleOptimismBench: Forecasting Bias and the Alignment Effect in Language Model JudgmentSeonglae Cho, Adriano Koshiyama
articleBias Audits Detect Bias but Disagree on Ranking: Evidence from Ten Instruments and Ten Frontier ModelsWilliam Guey, Pierrick Bougault, Wei Zhang, Vitor D. de Moura, Jos\'e O. Gomes