Apple Machine Learning Research (with Harvard University co-authors) · Weighted SSM Souping

Weighted State Souping for Efficient Long-Context Retrieval

This paper proposes weighted state souping, which learns query-dependent weights for combining cached state-space model (SSM) encodings of document, tool or memory chunks. The authors report substantial long-context gains across document question answering, needle-in-a-haystack retrieval, variable tracking and tool use. A sparse variant gives near-constant encoding cost with respect to context length, and the paper includes theoretical arguments on when souping works and when it may fail. The page gives no code or licence details.