aiqbench analysis
This research addresses a critical flaw in on-policy distillation where standard methods inadvertently teach students the teacher's own errors rather than just the capability gap. The proposed Cal-OPD framework isolates genuine improvement signals by filtering out teacher-specific deviations, resulting in superior performance across various model scales and mathematical reasoning benchmarks. This advancement offers a more robust training signal for scaling up reasoning models without inheriting the biases of the source model.
Calibrating Teacher--Student Discrepancy for On-Policy Distillation
arXiv:2609.21619v1 Announce Type: new Abstract: On-policy distillation (OPD) improves reasoning models by learning the token-level discrepancy between a stronger teacher and an on-policy student. However, this discrepancy does not purely reflect the capability gap between the teacher and the student: it also contains deviations arising from the teacher itself, which are consequently mixed into the observed teacher--student discrepancy and indiscriminately learned by standard OPD during training. This issue is further exacerbated by privileged OPD, where privileged information induces larger teacher-side likelihood shifts, thereby encouraging the student to learn more of the teacher's own deviation. We introduce \textbf{Calibrated On-Policy Distillation (Cal-OPD)}, which estimates the teacher's self-deviation region through positive and negative privileged interventions and calibrates the original teacher--student discrepancy by retaining only the component that lies beyond this region. Experiments on mathematical reasoning benchmarks show that, while retaining only about 52--65\% of the original teacher--student discrepancy as the optimization signal, Cal-OPD consistently outperforms standard OPD and its variants across model scales.
Abstract:On-policy distillation (OPD) improves reasoning models by learning the token-level discrepancy between a stronger teacher and an on-policy student. However, this discrepancy does not purely reflect the capability gap between the teacher and the student: it also contains deviations arising from the teacher itself, which are consequently mixed into the observed teacher--student discrepancy and indiscriminately learned by standard OPD during training. This issue is further exacerbated by privileged OPD, where privileged information induces larger teacher-side likelihood shifts, thereby encouraging the student to learn more of the teacher's own deviation. We introduce \textbf{Calibrated On-Policy Distillation (Cal-OPD)}, which estimates the teacher's self-deviation region through positive and negative privileged interventions and calibrates the original teacher--student discrepancy by retaining only the component that lies beyond this region. Experiments on mathematical reasoning benchmarks show that, while retaining only about 52--65\% of the original teacher--student discrepancy as the optimization signal, Cal-OPD consistently outperforms standard OPD and its variants across model scales.
Artificial Intelligence (cs.AI)
arXiv:2609.21619 [cs.AI]
(or arXiv:2609.21619v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.21619 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 249 words
Related coverage
SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillation
arXiv · Sep 14, 2026
Coupled Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback
arXiv · Sep 16, 2026
CV-OPD: Contrastive Vision On-policy Distillation
arXiv · Oct 12, 2026
Stochastic Teacher Intervention for Agentic On-Policy Distillation
arXiv · Oct 10, 2026