George Ling, Shanshan Zhong, Lijin Yang, Hao Yang, Nanxiang Li, Zhongzhan Huang (Carnegie Mellon University) · Benchmark Compression

Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand

The paper derives an analytical approximation to the error distribution when estimating full-benchmark scores from a small subset, and finds that random subsets usually give small errors but occasionally give large ones. It introduces BestRand, a compression method that keeps the low cost of random sampling while reducing the high-error tail. Validated on 70+ text, multimodal, and agent benchmarks, and extended to VLA evaluation, BestRand reduces high-error risk by 50.2% and estimation error by 16.4% on average, at 51.3% of the construction cost. Code and data are publicly available on GitHub.