FairStress: A deeper evaluation of fairness in LLM decision-making

The premise of FairStress, released by Ai2, is that the standard way of testing demographic fairness in decision-making models asks too little. That standard test swaps the demographic details of two candidates in an otherwise identical scenario and checks whether the model still picks the same one. The authors' objection is that matching answers can conceal a difference in how firmly each answer is held. Their benchmark therefore adds a second step: after the model chooses, it is given what the authors call an adversarial prefix — a persuasive statement arguing for the other candidate, drawing on authority, consensus, claimed evidence or legal concern — and the benchmark records whether the model switches. Each prefix comes with a matched version making the same kind of case for the opposite candidate, so the same argument can be tested in both directions. The stated target is what they call symmetric fairness: when the evidence supports a decision, the model should apply the same criteria and be equally hard to move regardless of which group benefits. The authors are explicit that this encodes one particular, equality-centered conception of fairness, and that an equity-based account treating historical disadvantage as decision-relevant would score things differently.

The construction explains what the resulting numbers can and cannot speak to. There are 300 two-candidate scenarios across 29 domains including hiring, lending, healthcare, education, housing and criminal justice; some have a candidate whose profile clearly better fits the stated criteria, while others are deliberately ambiguous so neither choice is better supported. Identity is varied in how explicitly it is conveyed, from absent, through a name or an indirect implication, to an explicit label. Combining those variations with the prefixes yields more than 13 million test cases, alongside a 50,000-item subset selected with item response theory to make evaluation cheaper. The authors report that with no demographic information in the prompt there was little difference in behavior, and that asymmetries grew as identity became more explicit: across the 15 models on the full benchmark, the average gap in abandoning a correct answer was 6.9 percentage points with implied identity and 9.3 points with an explicit label. They also report differences in how models worded their explanations even when they chose the same candidate in both versions, and note the risk that foregrounding identity in explaining a well-supported choice gives the candidate's qualifications less credit.

The result with the widest implications is methodological. Compared with BBQ, a widely used bias benchmark, FairStress pointed in a different direction, so the authors examined why. The standard BBQ setup they tested scores the probabilities of fixed answer options without the model generating a reply, whereas FairStress has the model write its own answer and choose. Holding the questions and model weights constant and varying the chat template, the question wording and whether answers were scored as fixed-option probabilities or generated, they found the measured direction of bias changed for three of four models tested, and moved to roughly neutral for the fourth. Their conclusion is that fixed-option scoring alone can miss behavior that appears in generated decisions, and that developers should look at both. Two further findings stay deliberately tentative. An audit of Olmo 3's training pipeline found that of 28,000 topic clusters in its fine-tuning and preference data, only nine concerned deciding on merit rather than demographics, and all nine were refusals — which the authors read as a gap between teaching a model to refuse discriminatory requests and showing it how to decide on the merits, while stating that whether this causes the behavioral gaps is an open question. And of nine debiasing interventions they tested, most did not reliably reduce the asymmetry and some traded accuracy heavily for balance — one activation-steering method dropped a model's accuracy on clear-cut decisions from 98% to 47% — with the closest to success being counterfactual chain-of-thought, which has the model reconsider the case with the identities swapped. An analysis of internal activations is included too, but the authors limit it to two models and a handful of contrasts and treat it as narrower than their behavioral results.