analiza aiqbench
This research addresses a critical data-collection bottleneck in on-policy self-distillation by introducing TISD, an algorithm that leverages teacher-student disagreement to force beneficial trajectory branches rather than just local corrections. The method demonstrates measurable gains in coding and science domains by ensuring the student model is trained on successor contexts privileged by the teacher but rarely sampled naturally. This advancement refines how large language models can be efficiently distilled without relying solely on standard rollout sampling strategies.
TISD: On-Policy Self-Distillation with Trajectory Intervention
arXiv:2609.30878v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) provides dense teacher targets, but evaluates them only along student-sampled rollouts. When the privileged teacher favors an alternative action at a visited prefix, OPSD can provide a target for the branch decision but cannot supervise the successor contexts induced by that action unless the student samples it. This creates a training-time data-collection bottleneck and suggests a different role for teacher-student disagreement: proposing a trajectory branch rather than identifying a sufficient local repair. Our diagnostic framework using controlled token interventions reveals that a teacher-preferred token at peak disagreement can improve student continuation success, while its local corrective value is limited. Motivated by this finding, we introduce a simple branch-regenerate-distill algorithm, Trajectory-Intervention Self-Distillation (TISD). TISD forces a teacher-selected branch action, returns suffix generation to the student, and distills the full trajectory under the privileged-context-conditioned teacher. Across the coding models, TISD improves average Avg@4 over SDPO by 1.2 percentage points. Across the science domains, it improves average Avg@128 by 0.8 points under an equal-step budget and by 0.3 points under an equal-time budget. These results support teacher-guided branching as a way to expose useful successor contexts for self-distillation.
Abstract:On-policy self-distillation (OPSD) provides dense teacher targets, but evaluates them only along student-sampled rollouts. When the privileged teacher favors an alternative action at a visited prefix, OPSD can provide a target for the branch decision but cannot supervise the successor contexts induced by that action unless the student samples it. This creates a training-time data-collection bottleneck and suggests a different role for teacher-student disagreement: proposing a trajectory branch rather than identifying a sufficient local repair. Our diagnostic framework using controlled token interventions reveals that a teacher-preferred token at peak disagreement can improve student continuation success, while its local corrective value is limited. Motivated by this finding, we introduce a simple branch-regenerate-distill algorithm, Trajectory-Intervention Self-Distillation (TISD). TISD forces a teacher-selected branch action, returns suffix generation to the student, and distills the full trajectory under the privileged-context-conditioned teacher. Across the coding models, TISD improves average Avg@4 over SDPO by 1.2 percentage points. Across the science domains, it improves average Avg@128 by 0.8 points under an equal-step budget and by 0.3 points under an equal-time budget. These results support teacher-guided branching as a way to expose useful successor contexts for self-distillation.
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2609.30878 [cs.AI]
(or arXiv:2609.30878v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.30878 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 285 cuvinte
Subiecte conexe
Continual learning might make your blocking monitors nearly useless
AI Alignment Forum · 24 sept. 2026
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · 16 oct. 2026
CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoning
arXiv · 26 sept. 2026
Interactive-Policy Distillation with Bidirectional Propose-and-Verify
arXiv · 30 sept. 2026