Ai2 (allenai), with researchers at Carnegie Mellon, NVIDIA and Johns Hopkins · XY problem benchmark
XYBench
XYBench is a benchmark of 8,115 questions from Stack Overflow, Stack Exchange, WikiHow and manual curation, testing whether assistants recognise misconceptions behind a user's stated request and steer them toward their real goal. It won the Best Resource Paper Award at EMNLP 2026. Across Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro, models were more likely to solve the stated request than the underlying goal, and focused mainly on the underlying goal in only 35–41% of Stack Overflow responses. The paper, data and code are linked, and the page does not state a licence.
Open the source ↗Similar Research & Safety releases
- One Table Benchmark
Yuanrui Wang, Kaifang Li, Xuanyue Li, Ziyun Li, Yan Lu, Xingxuan Zhang, Peng Cui (Stable AI) · Tabular Benchmark · Week 47, 2026
- Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages
Vladimir Dobrovolskii, Ibragim Badertdinov, Alexander Golubev (Nebius) · Long-tail coding benchmark · Week 47, 2026
- WebsiteBench
TIGER-AI-Lab · Web Agent Benchmark · Week 47, 2026
- Open ASR Leaderboard: Egyptian and Nigerian English splits
Hugging Face, with Liva AI · ASR Benchmark Update · Week 46, 2026
- StateBench
Ai2 (Allen Institute for AI) · State-tracking benchmark · Week 46, 2026
- Real-SWE
Specific Labs · Enterprise Coding Benchmark · Week 37, 2026
Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.