analiza aiqbench
This paper addresses optimization pathologies in semi-supervised learning by introducing the Pioneer Student, an auxiliary branch that decouples teacher and student parameters to prevent premature convergence. By operating in an independent parameter space before transferring knowledge back to the main model, this approach effectively mitigates regularization conflicts between labeled and unlabeled data. The method serves as a universal plug-and-play module that consistently enhances mainstream SSL techniques without altering their core architecture.
Decoupled Optimization for Teacher-Student Semi-Supervised Learning via a Pioneer Student
arXiv:2610.09609v1 Announce Type: new Abstract: Semi-supervised learning (SSL) relies on two core mechanisms: self-training under the Teacher-Student (T-S) framework and joint optimization of labeled and unlabeled losses. Despite their effectiveness, we find both mechanisms introduce distinct optimization pathologies. First, parameter coupling enforces strict synchronization between teacher and student, where strong regularization on the student degrades the teacher's fitting ability, thereby limiting the permissible generalization intensity. Second, the imbalance in gradient update consistency between labeled and unlabeled losses drives the shared parameters to prematurely converge to labeled-dominated local minima, creating a bottleneck for global optimization. To address both issues, we propose the Pioneer Student (PiS), an auxiliary branch that operates in an independent parameter space and periodically transfers accumulated knowledge back to the T-S model. Extensive experiments show that PiS is a universal plug-and-play module that consistently improves mainstream SSL methods.
Abstract:Semi-supervised learning (SSL) relies on two core mechanisms: self-training under the Teacher-Student (T-S) framework and joint optimization of labeled and unlabeled losses. Despite their effectiveness, we find both mechanisms introduce distinct optimization pathologies. First, parameter coupling enforces strict synchronization between teacher and student, where strong regularization on the student degrades the teacher's fitting ability, thereby limiting the permissible generalization intensity. Second, the imbalance in gradient update consistency between labeled and unlabeled losses drives the shared parameters to prematurely converge to labeled-dominated local minima, creating a bottleneck for global optimization. To address both issues, we propose the Pioneer Student (PiS), an auxiliary branch that operates in an independent parameter space and periodically transfers accumulated knowledge back to the T-S model. Extensive experiments show that PiS is a universal plug-and-play module that consistently improves mainstream SSL methods.
Machine Learning (cs.LG)
arXiv:2610.09609 [cs.LG]
(or arXiv:2610.09609v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.09609 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 229 cuvinte
Subiecte conexe
Why Ghost Outputs Teach: A Kernel-Based Understanding of Subliminal Learning
arXiv · 22 sept. 2026
Exploring Hierarchical Interest Representation For Meta Ads Deep Funnel Optimization
Meta Engineering · 15 iul. 2026
RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
arXiv · 18 sept. 2026
US-JEPA: A Joint Embedding Predictive Architecture for Ultrasound
arXiv · 9 sept. 2026