aiqbench analysis
TrustMOPD introduces a novel reliability-aware distillation framework that dynamically weights teacher inputs based on real-time RL displacement rather than static labels or randomization. This approach significantly improves recovery ratios in math, code, and instruction tasks by adapting supervision to the evolving generation context without requiring external ground truth. The method demonstrates that conditioning distillation targets on local model confidence is critical for leveraging complementary specialist knowledge effectively.
Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation
arXiv:2609.23697v1 Announce Type: new Abstract: Multi-teacher on-policy distillation allows a student to learn from complementary specialists on its own trajectories. Domain-routed approaches, however, select one teacher per example and keep it fixed throughout the response. This design both depends on labels that mixed training corpora often lack and cannot adapt teacher selection when the expertise required changes within a trajectory. We propose \textbf{TrustMOPD}, which replaces example-level teacher selection with label-free, token-level supervision allocation. At each student-generated prefix, TrustMOPD uses each specialist's RL-induced displacement from a shared pre-RL reference as a proxy for local reliability, calibrates these scores across teachers, and constructs a weighted distillation target. Across mathematics, code, and instruction following, TrustMOPD outperforms the strongest label-free baseline, increasing the recovery ratio from $54.4\%$ to $91.5\%$ on \textsc{SingleCap} and from $54.5\%$ to $98.0\%$ on \textsc{MultiCap}, while approaching label-based MOPD on \textsc{SingleCap}. Randomizing token-level weights independently of the student-generated prefix performs no better than uniform weighting, supporting the importance of conditioning supervision on the evolving generation context.
Abstract:Multi-teacher on-policy distillation allows a student to learn from complementary specialists on its own trajectories. Domain-routed approaches, however, select one teacher per example and keep it fixed throughout the response. This design both depends on labels that mixed training corpora often lack and cannot adapt teacher selection when the expertise required changes within a trajectory. We propose \textbf{TrustMOPD}, which replaces example-level teacher selection with label-free, token-level supervision allocation. At each student-generated prefix, TrustMOPD uses each specialist's RL-induced displacement from a shared pre-RL reference as a proxy for local reliability, calibrates these scores across teachers, and constructs a weighted distillation target. Across mathematics, code, and instruction following, TrustMOPD outperforms the strongest label-free baseline, increasing the recovery ratio from $54.4\%$ to $91.5\%$ on \textsc{SingleCap} and from $54.5\%$ to $98.0\%$ on \textsc{MultiCap}, while approaching label-based MOPD on \textsc{SingleCap}. Randomizing token-level weights independently of the student-generated prefix performs no better than uniform weighting, supporting the importance of conditioning supervision on the evolving generation context.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2609.23697 [cs.CL]
(or arXiv:2609.23697v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.23697 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 258 words
Related coverage
LiveLaw Supreme Court Half-Yearly Digest: January - June 2026 [Part 2]
livelaw.in · Sep 13, 2026
Trust Region On-Policy Distillation
arXiv · Jun 5, 2026
Multi-Turn On-Policy Distillation with Prefix Replay
arXiv · Jul 28, 2026
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · Oct 5, 2026