---
title: "XYBench: new 8,115-question benchmark measures whether models spot the user's real problem"
url: https://aibenchmark.gr/en/news/xybench-new-benchmark-8115-questions-models-real-user-problem/
language: en
published: 2026-11-16T16:19:25.354Z
updated: 2026-11-16T17:19:30.010Z
site: "AI Benchmark"
section: "Benchmarks"
---

# XYBench: new 8,115-question benchmark measures whether models spot the user's real problem

Hugging Face has unveiled XYBench, which tests whether five models identify the mistaken assumption behind a question. In the results, all of them fall short of humans at steering the user toward the real goal.

> Written with AI from the sources listed below. Editorial responsibility: Konstantinos Perros. How we write: https://aibenchmark.gr/en/how-we-write/

## Key facts

- XYBench contains 8,115 questions from Stack Overflow, Stack Exchange, hand-curated examples and WikiHow guides, and won the Best Resource Paper Award at EMNLP 2026. [1]
- On Stack Overflow, a solution for the stated request (Y) appeared in 75–86% of answers and a solution for the underlying goal (X) in 59–71%. [1]
- On the same set, the models focused mainly on the underlying goal in only 35–41% of answers and on the stated request in 47–57%. [1]
- Claude Sonnet 4.5 is used as the automated judge, with agreement with two human raters of 74.2–79.8% on emphasis and 84.6–93.3% on misconception identification. [1]

## What XYBench measures and how it was built

Hugging Face, in collaboration with researchers from Carnegie Mellon, NVIDIA and Johns Hopkins, has published XYBench. The benchmark is based on the so-called XY problem: the user wants to achieve X, assumes that approach Y will work, and asks for help with Y. In the example in the publication, someone asks how to firm up the icing that slides off their cupcakes, when the real problem is that the cupcakes are still warm.

The dataset consists of 8,115 questions: 1,557 from Stack Overflow, 184 from Stack Exchange, 102 hand-curated examples and 6,272 questions generated from WikiHow guides. For the questions from public forums, the researchers used the three highest-rated answers and, where one existed, the accepted answer, in order to determine the real goal. For WikiHow, Sonnet 4.5 generated questions built around plausible misconceptions, which were then enriched with context. Cases where answering the literal request was sufficient were removed.

## The three metrics and the automated judge

Each answer is evaluated on three metrics. "Pragmatic solution" records whether there is a solution for Y and for X, separately. "Pragmatic emphasis" shows whether the answer focuses on Y, on X, or balances the two. "Misconception identification" checks whether the answer directs the user toward X and avoids or discourages the mistaken Y.

The judge is Claude Sonnet 4.5, which classifies the individual sentences and assesses the emphasis. For validation, two humans independently evaluated 500 answers. Agreement with the judge, averaged across the two, ranged from 97.5–100% on whether a solution appears, 84.6–93.3% on misconception identification and 74.2–79.8% on emphasis.

## Results: models stay with the stated request

The models evaluated were Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro, with a single answer per question (single-turn). Human answers were evaluated separately on the same metrics. According to Hugging Face, across all four sources every model was more likely to include a solution for Y than for X. On Stack Overflow, the rates were 75–86% for Y and 59–71% for X. As for emphasis, 35–41% of answers focused mainly on X and 47–57% on Y; the rest gave similar weight to both.

In an "oracle" experiment, where both Y and X were provided along with the question, Sonnet 4.6's Stack Overflow rate for a solution to the underlying goal rose from 59% to 67%, while its misconception identification rate fell from 47% to 34%. On the sources with human reference answers, every model again fell short of humans at identifying the misconception.

In a second experiment, six models (the five plus GPT-4.1) judged two candidate answers per question: one aimed at the literal request and one that explained the misconception. On Stack Overflow they chose the goal-focused answer in 71–84% of cases. The comparison with the 35–41% in their own answers shows, according to the authors, a gap between recognizing and producing useful advice.

## Limits and next steps

The publication comes from the benchmark's own creators, and the judge is a model, with the lowest agreement with humans on the emphasis metric. XYBench currently covers English-language queries and mainly procedural problems. Extension to other languages, cultures and conversational settings remains open.

The authors propose the benchmark as a way to test whether changes in prompting or training improve misconception identification, the provision of solutions for the underlying goal and the shifting of the answer toward it. Future work could examine which training methods help models act on inferred intent.

## Sources

1. Hugging Face: [XYBench: Helping with the problem behind the question](https://huggingface.co/blog/allenai/xybench)
