Peter Gostev works on AI capability at Arena and created BullshitBench, a benchmark of whether language models challenge nonsensical premises. His project examines a specific failure mode: giving a polished answer when the question itself should be rejected.[2][3]
Start with BullshitBench →
X · petergostev Contributions
BullshitBench distinguishes clear pushback, partial challenge and acceptance of nonsense. Its first version contains 55 questions; the second contains 100. Three model judges classify responses, and the reported clear-pushback rate excludes candidate refusals from its denominator. Those scoring choices matter when comparing a model that challenges a premise with one that simply declines to answer.[3]
Ideas
The benchmark publishes prompts, results and scoring rules. It measures responses to deliberately broken questions rather than general factual accuracy. Comparing the categories makes the underlying judgment visible: a response can notice something wrong yet still proceed with the requested explanation.[3]
Tools, writing & talks
Tools
Start here
BullshitBench
github.comQuestions and scoring rules for detecting whether models push back on nonsensical premises.
Read the question examples and refusal policy before comparing scores. Browsing results requires no setup; reproducing runs requires configured model APIs.
Creator, explicitly named in README[3]Original source