Apple Machine Learning Research (Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald, Federico Danieli) · Alignment limits theory

Theoretical Limits of Language Model Alignment

Apple researchers derive the information-theoretic limits of KL-regularised language model alignment, giving a closed-form maximum reward gain for a fixed KL budget. The paper proposes a practical estimator that predicts achievable alignment gains from base model samples alone, and shows that reward ensembling mitigates reward hacking. Empirically, on safety and summarisation tasks, best-of-N closely approaches the theoretical limit while PPO and GRPO remain substantially suboptimal. The paper is listed as a NeurIPS publication.