analiza aiqbench
This study isolates the impact of privileged context design on policy drift within on-policy self-distillation (OPSD), demonstrating that content variations induce significantly larger shifts in model geometry than source variations. By quantifying these effects through KL divergence and parameter-update alignment, the authors provide actionable guidance for stabilizing continual learning pipelines without compromising prior-task retention. The findings highlight that how information is presented during distillation is a critical lever for controlling trajectory stability in large language models.
Privileged Context as Drift in On-Policy Self-Distillation
arXiv:2610.07842v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) trains a language model to match a copy of itself conditioned on privileged context. Existing work varies what privileged context contains and how it is produced while also changing models, data, and training setups, making the effects of privileged context design difficult to isolate. Motivated by efforts in continual learning to reduce catastrophic forgetting, we study how the choice of privileged context affects policy drift. Specifically, we vary two axes: content (a demonstration, feedback, or rephrase) and source (external, self-generated with a verifier, or self-generated without a verifier). We train Qwen2.5-7B with OPSD across these nine combinations and three datasets, measuring target-task accuracy, prior-task retention, reverse KL from the base policy, and parameter-update geometry. Holding source fixed, changing content spans a wider median KL range than holding content fixed and changing source. The ratio between these ranges is $5.1\times$ for per-token KL and $2.2\times$ for per-sequence KL. Parameter-update geometry shows the same pattern: updates from adapters that share content are more closely aligned (mean cosine $0.571$) than updates from adapters that share source ($0.255$). For continual learning, these findings suggest that privileged context should be treated as part of OPSD's stability design because it is associated with how far and in what direction the policy moves.
Abstract:On-policy self-distillation (OPSD) trains a language model to match a copy of itself conditioned on privileged context. Existing work varies what privileged context contains and how it is produced while also changing models, data, and training setups, making the effects of privileged context design difficult to isolate. Motivated by efforts in continual learning to reduce catastrophic forgetting, we study how the choice of privileged context affects policy drift. Specifically, we vary two axes: content (a demonstration, feedback, or rephrase) and source (external, self-generated with a verifier, or self-generated without a verifier). We train Qwen2.5-7B with OPSD across these nine combinations and three datasets, measuring target-task accuracy, prior-task retention, reverse KL from the base policy, and parameter-update geometry. Holding source fixed, changing content spans a wider median KL range than holding content fixed and changing source. The ratio between these ranges is $5.1\times$ for per-token KL and $2.2\times$ for per-sequence KL. Parameter-update geometry shows the same pattern: updates from adapters that share content are more closely aligned (mean cosine $0.571$) than updates from adapters that share source ($0.255$). For continual learning, these findings suggest that privileged context should be treated as part of OPSD's stability design because it is associated with how far and in what direction the policy moves.
Machine Learning (cs.LG)
arXiv:2610.07842 [cs.LG]
(or arXiv:2610.07842v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.07842 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 303 cuvinte
Subiecte conexe
RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
arXiv · 12 iun. 2026
RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
arXiv · 15 sept. 2026
Geometric Self-Distillation for Reasoning Generalization
arXiv · 6 oct. 2026
One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context
arXiv · 22 iul. 2026