aiqbench analysis
DuoOPD introduces a novel framework for on-policy distillation that dynamically adjusts feedback direction and teacher support based on joint teacher-student outcomes, addressing inefficiencies in existing methods where incorrect teacher guidance degrades student performance. By leveraging a unified rule to handle all outcome combinations across diverse tasks like code generation and scientific calculation, the method achieves significant accuracy improvements over current baselines without requiring task-specific tuning. This advancement represents a meaningful step forward in optimizing multi-task learning dynamics for large language models.
DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillation
arXiv:2609.33711v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student on its own responses with token-level feedback from a stronger teacher, yet the teacher can fail on questions the student already answers correctly, and how often each model succeeds varies across tasks. OPD ignores these outcomes and, on average, pushes down even the student's correct responses; gating feedback by student correctness fixes the direction but uses the teacher in the same way whether or not it succeeded. We introduce DuoOPD, in which the student's outcome sets the direction of feedback and the joint teacher-student outcome decides how the teacher supports it: when only the teacher succeeds, its verified answer becomes context for scoring the student's failed response, and when only the student succeeds, a weight shared within the task reinforces the whole response. A single rule covers all four outcome combinations without task-specific settings. Across Qwen3 and Llama, DuoOPD outperforms all five baselines in mean macro accuracy, improving over OPD by 2.58 and 5.98 percentage points, and it also leads on two further task mixtures spanning scientific calculation, instruction following, and code generation. Ablations show that outcome-based direction alone stays near the gated baseline, while the joint-outcome designs supply most of the gain.
Abstract:On-policy distillation (OPD) trains a student on its own responses with token-level feedback from a stronger teacher, yet the teacher can fail on questions the student already answers correctly, and how often each model succeeds varies across tasks. OPD ignores these outcomes and, on average, pushes down even the student's correct responses; gating feedback by student correctness fixes the direction but uses the teacher in the same way whether or not it succeeded. We introduce DuoOPD, in which the student's outcome sets the direction of feedback and the joint teacher-student outcome decides how the teacher supports it: when only the teacher succeeds, its verified answer becomes context for scoring the student's failed response, and when only the student succeeds, a weight shared within the task reinforces the whole response. A single rule covers all four outcome combinations without task-specific settings. Across Qwen3 and Llama, DuoOPD outperforms all five baselines in mean macro accuracy, improving over OPD by 2.58 and 5.98 percentage points, and it also leads on two further task mixtures spanning scientific calculation, instruction following, and code generation. Ablations show that outcome-based direction alone stays near the gated baseline, while the joint-outcome designs supply most of the gain.
Machine Learning (cs.LG)
arXiv:2609.33711 [cs.LG]
(or arXiv:2609.33711v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.33711 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 293 words
Related coverage
TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL
arXiv · Sep 30, 2026
From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Search
arXiv · Jul 28, 2026
Frontier post-training recipe review with Finbarr Timbers
Interconnects · Jun 16, 2026
MAS-OPD: On-Policy Distillation for Multi-agent Systems
arXiv · Sep 30, 2026