Ai2 (allenai), with researchers at Carnegie Mellon, NVIDIA and Johns Hopkins · XY problem benchmark

XYBench

XYBench is a benchmark of 8,115 questions from Stack Overflow, Stack Exchange, WikiHow and manual curation, testing whether assistants recognise misconceptions behind a user's stated request and steer them toward their real goal. It won the Best Resource Paper Award at EMNLP 2026. Across Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro, models were more likely to solve the stated request than the underlying goal, and focused mainly on the underlying goal in only 35–41% of Stack Overflow responses. The paper, data and code are linked, and the page does not state a licence.

Open the source ↗

Similar Research & Safety releases

Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.