aiqbench analysis
This research investigates the failure modes of on-policy distillation in language models, attributing early loss plateaus to a collapse in gradient-based learning signals despite significant remaining loss. The authors provide theoretical guarantees for recovery when teacher-student parameterization is sufficiently close and hypothesize that limited representation adaptation contributes to this signal degradation. These findings offer critical insights into the stability limits of capability transfer methods without relying on reinforcement learning.
Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals
arXiv:2610.11247v1 Announce Type: cross Abstract: On-policy distillation (OPD) enables effective capability transfer between language models, yet the mechanisms underlying its failures are not fully understood. Across code generation and mathematical reasoning, OPD with larger-scale teachers exhibits early loss plateaus, with an average final loss reduction of 25.1% after 200 updates, compared with 96.2% for self-RL teachers, obtained by further reinforcement learning (RL) training of the initial student. To understand this difference, we analyze OPD as an idealized continuous-time dynamical system in the small-learning-rate limit. Our training-log diagnostics associate these plateaus with an early decline in a gradient-based learning-signal proxy while substantial loss remains; these measurements do not establish why the underlying gradient weakens. We further prove a local recovery guarantee for teachers sufficiently close to the initial student in a shared parameterization under regularity conditions, offering a conditional explanation for the success of self-RL teachers in our experiments. Across runs with and without loss plateaus, we observe small relative parameter changes (0.025-0.098%) and high similarity between the student's representations before and after OPD (linear CKA $>0.98$ across layers). These observations suggest that limited representation adaptation may contribute to learning-signal collapse, a hypothesis that remains to be tested. Code is available at https://github.com/leizhao7/opd-learning-signals.
Abstract:On-policy distillation (OPD) enables effective capability transfer between language models, yet the mechanisms underlying its failures are not fully understood. Across code generation and mathematical reasoning, OPD with larger-scale teachers exhibits early loss plateaus, with an average final loss reduction of 25.1% after 200 updates, compared with 96.2% for self-RL teachers, obtained by further reinforcement learning (RL) training of the initial student. To understand this difference, we analyze OPD as an idealized continuous-time dynamical system in the small-learning-rate limit. Our training-log diagnostics associate these plateaus with an early decline in a gradient-based learning-signal proxy while substantial loss remains; these measurements do not establish why the underlying gradient weakens. We further prove a local recovery guarantee for teachers sufficiently close to the initial student in a shared parameterization under regularity conditions, offering a conditional explanation for the success of self-RL teachers in our experiments. Across runs with and without loss plateaus, we observe small relative parameter changes (0.025-0.098%) and high similarity between the student's representations before and after OPD (linear CKA $>0.98$ across layers). These observations suggest that limited representation adaptation may contribute to learning-signal collapse, a hypothesis that remains to be tested. Code is available at this https URL.
50 pages. Code: this https URL
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2610.11247 [cs.LG]
(or arXiv:2610.11247v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.11247 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 303 words
Related coverage
SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
arXiv · Oct 1, 2026
Why I Left Google DeepMind
LessWrong · Jul 15, 2026
Links #3: 2026/06 Part 1
LessWrong · Jun 15, 2026
Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight
arXiv · Oct 7, 2026