Forcing a Letter Suppresses Refusal: Mechanistic Evidence from WMDP

WMDP (Weapons of Mass Destruction Proxy) is a benchmark of 3,668 four-option multiple-choice questions about hazardous biology, chemistry and cybersecurity knowledge, and safety evaluations often cite scores like it as evidence about how risky a model is. A preprint from researchers at HSE University, ITMO University and AIRI argues that the multiple-choice format hides whether a model's refusal behaviour works at all. When two small open models, Meta's Llama-3.1-8B and Qwen's Qwen3-8B, saw WMDP questions rewritten without answer options, they sometimes refused: about one sampled answer in ten for Llama, and far fewer for Qwen. Given the same questions as multiple choice, with an instruction to reply with only a letter, they almost never refused. They also picked the correct letter about as often on questions they usually refused in open form as on questions they never refused. The authors conclude that a multiple-choice score measures whether the model has the knowledge, not whether it would decline to share it when asked openly.

To explain the gap, the team looked inside the models. A tuned lens is a tool that estimates which word a model is leaning toward at each internal layer. On open-ended questions, the probability of the word that typically opens a refusal ("I", as in "I cannot") rose gradually from around layer 11, and rose further on questions the model usually refused. On multiple-choice questions, the model instead committed sharply to an answer letter in a narrow band of middle layers (15 to 18 in Llama, 22 to 25 in Qwen), and the refusal word stayed near zero throughout. A second measure compares the model's internal state with a refusal direction, a pattern of activity that earlier research found to mediate refusals. In Llama, the open-ended versions of questions it usually refused aligned with that direction about three times as strongly as the same questions in multiple-choice form, which sat close to the level of a harmless control.

The strongest evidence comes from switching parts of the model off. Disabling the ten attention heads, small internal components, that contributed most to producing the answer letter lowered Llama's probability of choosing the correct letter from about 0.77 to about 0.65 and raised the probability of the refusal word from under 0.01 to about 0.05. Disabling the same number of randomly chosen heads in the same layers left both close to where they started. The authors then split the multiple-choice prompt into parts and found that the instruction to answer with only a letter does the suppressing. On questions Llama usually refused, adding that instruction cut the refusal word's probability from about 0.5 to about 0.05, whether it appeared in the system prompt or the user's message, while a sentence that merely framed the item as a multiple-choice question left substantial refusal in place.

The evidence is narrower than the headline. Both models have about 8 billion parameters, and the authors say it is unknown whether the pattern holds in larger ones. The causal result was established only on Llama: in Qwen, disabling the same kind of heads did not bring refusal back, and the refusal-direction signal was weak and based on small samples. All results come from one benchmark. Within those limits, the paper gives a mechanism for an effect earlier behavioural studies had reported, and its practical conclusion is the authors' own recommendation: a safety assessment that cites WMDP-style multiple-choice scores should pair them with open-ended testing, because the letter-only format can leave the refusal mechanism untested. The paper is on arXiv. A separate preprint covered on VibeLeaderboard, There Is No Neutral Harness, found that option order, prompt wording and the way an answer is read off also move models' scores on multiple-choice benchmarks such as MMLU; this paper raises the same concern about format for safety behaviour, and traces it inside the model.