analiza aiqbench
This research introduces Decoupled Credit Self-Distillation (DCSD) to resolve the coupling between credit direction and magnitude in reinforcement learning from human feedback, thereby reducing reliance on error-prone teacher supervision. By separating these signals, the method achieves significant performance gains on mathematical and multimodal reasoning benchmarks while improving the reliability of step-level policy optimization.
Can We Trust the Teacher? Decoupled Credit Direction-Magnitude for Self-Distillation
arXiv:2609.34848v1 Announce Type: new Abstract: RLVR provides reliable trajectory-level credit, while OPSD offers dense supervision for token-level credit. This exposes a fundamental coupling when updating step-level credit direction and magnitude with teacher supervision, preventing steps from receiving reliable credit directions and contribution magnitudes, while making both vulnerable to teacher judgment errors and preference variance, as supported by our theoretical analysis. To separate credit direction from its contribution magnitude, we introduce \textit{Decoupled Credit Self-Distillation (DCSD)}, which theoretically decouples credit direction and magnitude into two reliable signals and uses them to calibrate privileged teacher supervision. Specifically, we design belief-margin probing to determine credit direction and marginal information gain to quantify credit magnitude, enabling step-to-token credit assignment for policy optimization. Across 11 benchmarks, DCSD achieves the best overall scores against GRPO, OPSD, RLSD, and RLCSD. Compared with base models, DCSD improves the overall score by 8.45 points on mathematical reasoning and 7.01 points on multimodal reasoning, while correcting the credit direction for 6\% of tokens and yielding a 1.5$\times$ reduction in token credit magnitude.
Title:Can We Trust the Teacher? Decoupled Credit Direction-Magnitude for Self-Distillation
Abstract:RLVR provides reliable trajectory-level credit, while OPSD offers dense supervision for token-level credit. This exposes a fundamental coupling when updating step-level credit direction and magnitude with teacher supervision, preventing steps from receiving reliable credit directions and contribution magnitudes, while making both vulnerable to teacher judgment errors and preference variance, as supported by our theoretical analysis. To separate credit direction from its contribution magnitude, we introduce \textit{Decoupled Credit Self-Distillation (DCSD)}, which theoretically decouples credit direction and magnitude into two reliable signals and uses them to calibrate privileged teacher supervision. Specifically, we design belief-margin probing to determine credit direction and marginal information gain to quantify credit magnitude, enabling step-to-token credit assignment for policy optimization. Across 11 benchmarks, DCSD achieves the best overall scores against GRPO, OPSD, RLSD, and RLCSD. Compared with base models, DCSD improves the overall score by 8.45 points on mathematical reasoning and 7.01 points on multimodal reasoning, while correcting the credit direction for 6\% of tokens and yielding a 1.5$\times$ reduction in token credit magnitude.
Artificial Intelligence (cs.AI)
arXiv:2609.34848 [cs.AI]
(or arXiv:2609.34848v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.34848 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 270 cuvinte
Subiecte conexe
Microsoft’s AI chief says superintelligence is near, but won’t take your job
The Verge · 8 iun. 2026
H$^2$SD: Hybrid Hindsight Self-Distillation
arXiv · 22 iul. 2026
RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution
arXiv · 14 oct. 2026
StepOPSD: Step-Aware Online Preference Self-Distillation for Agent Reinforcement Learning
arXiv · 27 aug. 2026