aiqbench analysis
This research identifies how On-Policy Distillation (OPD) propagates specific teacher errors, revealing a stark divergence where forward-KL divergences transmit mistakes significantly while reverse-KL variants largely suppress them. The study demonstrates that even minor training budget increases can cause models to adopt these inherited errors if the KL direction permits it, necessitating audits on sampled outputs rather than relying solely on greedy evaluation.
ErrOPD: When Does On-Policy Distillation Transfer Teacher Errors?
arXiv:2605.08737v2 Announce Type: replace-cross Abstract: A distilled student can copy its teacher's mistakes along with its skills. On-policy distillation (OPD), used in recent post-training recipes, has the teacher grade every token of the student's own outputs. Which teacher errors does it pass on? We build ErrOPD, a testbed whose teachers are fine-tuned to close a JSON list one item early, an error their students almost never make. The answer depends on the KL direction: across four model families, forward-KL OPD transmits the error within the training budget (+13-23 points), while reverse-KL OPD at our reference learning rate adds under 0.05 points. A duplicated ID and, where the decision arises, two SQL errors show the same split. The reverse-KL update scales with the student's own probability of a token, and an intervention tests this: giving reverse KL forward KL's gradient magnitude at the error decision alone makes it transmit on two model pairs. Reverse-KL OPD can delay such errors rather than block them: some runs adopt the error with longer training, and Qwen3 students that already make it, even rarely, adopt it within the budget. Because greedy evaluation on Qwen3 misses the inherited error, distilled students should also be audited on sampled outputs.
Title:ErrOPD: When Does On-Policy Distillation Transfer Teacher Errors?
Abstract:A distilled student can copy its teacher's mistakes along with its skills. On-policy distillation (OPD), used in recent post-training recipes, has the teacher grade every token of the student's own outputs. Which teacher errors does it pass on? We build ErrOPD, a testbed whose teachers are fine-tuned to close a JSON list one item early, an error their students almost never make. The answer depends on the KL direction: across four model families, forward-KL OPD transmits the error within the training budget (+13-23 points), while reverse-KL OPD at our reference learning rate adds under 0.05 points. A duplicated ID and, where the decision arises, two SQL errors show the same split. The reverse-KL update scales with the student's own probability of a token, and an intervention tests this: giving reverse KL forward KL's gradient magnitude at the error decision alone makes it transmit on two model pairs. Reverse-KL OPD can delay such errors rather than block them: some runs adopt the error with longer training, and Qwen3 students that already make it, even rarely, adopt it within the budget. Because greedy evaluation on Qwen3 misses the inherited error, distilled students should also be audited on sampled outputs.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2605.08737 [cs.LG]
(or arXiv:2605.08737v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.08737 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 301 words
Related coverage
Coupled Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback
arXiv · Sep 16, 2026
Beyond Imitation: Reflective On-Policy Self-Distillation for LLM Reasoning
arXiv · Sep 30, 2026
OVD: On-policy Verbal Distillation
arXiv · Sep 30, 2026
Rethinking Classifier-Free Guidance in On-Policy Diffusion Distillation
arXiv · Jul 28, 2026