What XYBench measures and how it was built
Hugging Face, in collaboration with researchers from Carnegie Mellon, NVIDIA and Johns Hopkins, has published XYBench. The benchmark is based on the so-called XY problem: the user wants to achieve X, assumes that approach Y will work, and asks for help with Y. In the example in the publication, someone asks how to firm up the icing that slides off their cupcakes, when the real problem is that the cupcakes are still warm.
The dataset consists of 8,115 questions: 1,557 from Stack Overflow, 184 from Stack Exchange, 102 hand-curated examples and 6,272 questions generated from WikiHow guides. For the questions from public forums, the researchers used the three highest-rated answers and, where one existed, the accepted answer, in order to determine the real goal. For WikiHow, Sonnet 4.5 generated questions built around plausible misconceptions, which were then enriched with context. Cases where answering the literal request was sufficient were removed.
The three metrics and the automated judge
Each answer is evaluated on three metrics. "Pragmatic solution" records whether there is a solution for Y and for X, separately. "Pragmatic emphasis" shows whether the answer focuses on Y, on X, or balances the two. "Misconception identification" checks whether the answer directs the user toward X and avoids or discourages the mistaken Y.
The judge is Claude Sonnet 4.5, which classifies the individual sentences and assesses the emphasis. For validation, two humans independently evaluated 500 answers. Agreement with the judge, averaged across the two, ranged from 97.5–100% on whether a solution appears, 84.6–93.3% on misconception identification and 74.2–79.8% on emphasis.
Results: models stay with the stated request
The models evaluated were Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro, with a single answer per question (single-turn). Human answers were evaluated separately on the same metrics. According to Hugging Face, across all four sources every model was more likely to include a solution for Y than for X. On Stack Overflow, the rates were 75–86% for Y and 59–71% for X. As for emphasis, 35–41% of answers focused mainly on X and 47–57% on Y; the rest gave similar weight to both.
In an "oracle" experiment, where both Y and X were provided along with the question, Sonnet 4.6's Stack Overflow rate for a solution to the underlying goal rose from 59% to 67%, while its misconception identification rate fell from 47% to 34%. On the sources with human reference answers, every model again fell short of humans at identifying the misconception.
In a second experiment, six models (the five plus GPT-4.1) judged two candidate answers per question: one aimed at the literal request and one that explained the misconception. On Stack Overflow they chose the goal-focused answer in 71–84% of cases. The comparison with the 35–41% in their own answers shows, according to the authors, a gap between recognizing and producing useful advice.
Limits and next steps
The publication comes from the benchmark's own creators, and the judge is a model, with the lowest agreement with humans on the emphasis metric. XYBench currently covers English-language queries and mainly procedural problems. Extension to other languages, cultures and conversational settings remains open.
The authors propose the benchmark as a way to test whether changes in prompting or training improve misconception identification, the provision of solutions for the underlying goal and the shifting of the answer toward it. Future work could examine which training methods help models act on inferred intent.