XYBench: Helping with the problem behind the question
Source
allenai.org
Date
Why it matters
XYBench measures whether assistants spot a flawed premise and redirect to the real goal, a failure common in coding help. Use it to evaluate agents that follow misguided requests literally.
Key takeaways · AI-distilled
XYBench's 8,115 questions come from Stack Overflow (1,557), Stack Exchange (184), 102 hand-curated cases and 6,272 WikiHow-based questions generated with Sonnet 4.5, each labeled with the stated request and the inferred underlying goal.
On Stack Overflow questions, models mentioned a fix for the real goal in 59-71% of answers but made it the focus in only 35-41%, against 55% for top human answers. Humans steered users off the mistaken approach 79% of the time; models did so 47-60%.
Telling the model the user's real goal did not close the gap: Sonnet 4.6's goal-solution rate rose from 59% to 67%, but its misconception-identification rate fell from 47% to 34%, and every model still trailed humans on that measure.
Shown two answers, models picked the goal-focused one 71-84% of the time, yet only 35-41% of their own answers focused on the goal. Ai2 calls this a gap between recognizing useful advice and producing it that a multiple-choice test alone would miss.
Sonnet 4.5 is the automated judge. Against two human reviewers on 500 responses, agreement was 97.5-100% on whether a solution appeared but 74.2-79.8% on emphasis. The benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → covers English and mainly procedural problems.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
