Junhao Shen, Qing Zong, Yue Guo, Zhanqiu Zhang, Yiwen Guo, Hong Cheng (The Chinese University of Hong Kong, Database Group) · Decision model benchmark

MMDM: Measuring Massive Multitask Decision Making

MMDM is a benchmark of 5,448 questions across 16 task categories for single-select, multi-select and soft-label decision making, covering agent state tracking, tool and skill selection, preference judgments and rubric scoring. Option spaces range from 2 to 1,024 and inputs reach 64,817 tokens. The authors evaluate 28 decision models and find they handle common knowledge well but struggle with interdependent options, constrained decisions and multi-select tasks, with the best Hard Multi-select Accuracy at 36.81%.