Bias Audits Detect Bias but Disagree on Ranking: Evidence from Ten Instruments and Ten Frontier Models
Source
William Guey, Pierrick Bougault, Wei Zhang, Vitor D. de Moura, Jos\'e O. Gomes
Author
William Guey, Pierrick Bougault, Wei Zhang, Vitor D. de Moura, Jos\'e O. Gomes
Date
Why it matters
Ten bias-audit tools run against the same ten frontier models all detect bias, but their rankings of which model is more biased agree no better than chance, a warning against using any single audit score to rank models for compliance.
articleEvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language ModelsXinning Li, Kemunto Ochwang'i, Aryasomayajula Ram Bharadwaj, Alexandra Souly, Robert Kirk
articleWho Do Language Models Think Is Competent? A Mechanistic Analysis of Occupational BiasKeren Fuentes, Aaron Mueller
articleThe Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the MeanYonghong Zhang, Shadi Motaali, Vu Phong Dinh, Avin Piroutiniya, Jorge E. L\'opez de Vergara, Luis de Pedro, Ricardo Correia, Isabel M. Parra, Yong Xie