Ai2 (Allen Institute for AI) · State-tracking benchmark

StateBench

StateBench is a long-context benchmark that tests how well language models track changing entity state across long documents, at four difficulty levels from immutable assignment to assignment by reference. Ai2 reports that pretrained transformers handle the simplest level but stay at chance on general finite problems, while Gated DeltaNet and hybrid models trained directly on the task generalised almost perfectly on the easier levels out to 64,000 assignments. Four frontier models performed well on the easier levels but fell to 13% or lower accuracy on the longest general finite problems. A tech report, dataset and code are linked from the post; the licence is not stated.