analiza aiqbench
This research introduces Prefix-Guided On-Policy Distillation (PG-OPD), a method that optimizes training efficiency for long-horizon reasoning models by pruning low-compatibility trajectories early. By leveraging initial prefix overlap as a proxy for future value, the approach reduces computational waste while achieving significant accuracy gains on math benchmarks like AMC and AIME. The technique offers a practical pathway to scale distillation without incurring prohibitive costs associated with full rollout budgets for all candidates.
Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts
arXiv:2606.21994v2 Announce Type: replace Abstract: On-policy distillation (OPD) improves reasoning models by applying dense teacher supervision on student-sampled trajectories. However, scaling OPD to long-horizon reasoning exposes a reliability and efficiency problem: standard OPD assigns every candidate the same long rollout budget, even though some trajectories may quickly become weakly aligned with the teacher and provide less useful supervision. Prior analyses suggest that teacher--student compatibility is important for OPD success, motivating early-prefix top-k overlap as a proxy for continuation value. Continuing low-overlap trajectories to the full rollout length may cause them to drift further from the teacher as generation proceeds, increasing computational cost while providing limited distillation benefit. To address this, we introduce Prefix-Guided On-Policy Distillation (PG-OPD), a rollout-allocation framework that estimates continuation value from fixed-length prefixes. PG-OPD computes teacher-student top-k overlap in an early probe window and allocates long rollouts only to high-overlap candidates, while stopping the rest at the prefix length. Across teacher--student combinations on AMC, AIME, and HMMT benchmarks, PG-OPD achieves up to a 4.80-point accuracy gain and up to a 2.46x wall-clock speedup across configurations. These results demonstrate that using early-prefix compatibility to guide candidate pruning can improve both training efficiency and reasoning performance.
Abstract:On-policy distillation (OPD) improves reasoning models by applying dense teacher supervision on student-sampled trajectories. However, scaling OPD to long-horizon reasoning exposes a reliability and efficiency problem: standard OPD assigns every candidate the same long rollout budget, even though some trajectories may quickly become weakly aligned with the teacher and provide less useful supervision. Prior analyses suggest that teacher--student compatibility is important for OPD success, motivating early-prefix top-k overlap as a proxy for continuation value. Continuing low-overlap trajectories to the full rollout length may cause them to drift further from the teacher as generation proceeds, increasing computational cost while providing limited distillation benefit. To address this, we introduce Prefix-Guided On-Policy Distillation (PG-OPD), a rollout-allocation framework that estimates continuation value from fixed-length prefixes. PG-OPD computes teacher-student top-k overlap in an early probe window and allocates long rollouts only to high-overlap candidates, while stopping the rest at the prefix length. Across teacher--student combinations on AMC, AIME, and HMMT benchmarks, PG-OPD achieves up to a 4.80-point accuracy gain and up to a 2.46x wall-clock speedup across configurations. These results demonstrate that using early-prefix compatibility to guide candidate pruning can improve both training efficiency and reasoning performance.
Machine Learning (cs.LG)
arXiv:2606.21994 [cs.LG]
(or arXiv:2606.21994v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.21994 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 284 cuvinte
Subiecte conexe
Learning from Repaired Reasoning: Root-Cause-Guided On-Policy Distillation
arXiv · 5 oct. 2026
REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse
arXiv · 30 sept. 2026
How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation
arXiv · 6 oct. 2026
OPTD: On-Policy Transition Distillation with Consistency-Guided Adaptive Compression for Few-Step Diffusion Language Models
arXiv · 6 aug. 2026