Apple Machine Learning Research (with Harvard University authors) · Gating in SSMs

On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models

This NeurIPS paper studies how the gating mechanism in state space models (SSMs) affects their training and generalisation. The authors argue, through theory and experiments, that gating causes SSMs to first learn an in-weights memorisation solution, which can delay or prevent convergence to a correct in-context learning solution, even without architectural or memory-capacity limits. They also find gating often helps generalisation to longer sequences. The page gives no code or licence details.