You can rerun a paired-prompt bias eval on your own models, using the released prompts and grader prompts, instead of building a political-neutrality test from scratch.
Key takeaways · AI-distilled
Anthropic's Paired Prompts method asks a model for help on the same contested topic from two opposing sides, then grades the pair on even-handedness, acknowledgement of opposing perspectives, and refusals.
The automated version ran 1,350 prompt pairs across 9 task types and 150 topics. Anthropic reports even-handedness of 95% for Opus 4.1 and 94% for Sonnet 4.5, near Gemini 2.5 Pro (97%) and Grok 4 (96%), above GPT-5 (89%) and Llama 4 (66%).
Sonnet 4.5 served as grader. As a check, it agreed with GPT-5 on 92% of samples for even-handedness, while human graders in a similar test agreed only 85% of the time, which Anthropic reads as models grading more consistently than people.
Anthropic shapes the behavior two ways: Claude.ai system promptThe standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.Full definition → instructions, which it says are not foolproof, and character training that uses reinforcement learning to reward responses closer to traits such as not being identifiable as conservative or liberal.
Stated limits: the evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → covers mostly US political discourse, single-turn exchanges only, and weights all topics equally. Anthropic says other bias measures could give quite different results, and API users can configure Claude to reflect their own values.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
