StepFun (Sinuo Wang et al.) · Parameter-prior distillation

P^2-MOPD: Accelerating Multi-Teacher On-Policy Distillation via Parameter Priors

P^2-MOPD treats a linear combination of student and expert parameters as one differentiable model, and jointly optimises the mixing coefficients and the student weights under the on-policy distillation objective. Unlike task arithmetic, the student stays trainable, so the experts act as parameter priors. Across four domains at 1.7B and 4B scales, the 1.7B model reaches 94.7% average recovery versus 80.9% for MOPD, and surpasses MOPD's final average recovery in half the training steps.