analiza aiqbench
This research demonstrates that training a strong student model on imperfect labels from a weak teacher can fundamentally alter the scaling laws of test error, enabling optimal performance even when the teacher fails to improve with sample size. By deriving deterministic equivalents for random feature ridge regression, the study identifies specific regimes where this weak-to-strong generalization improves both bias-dominated and variance-dominated settings. The finding is significant for practitioners relying on iterative self-training pipelines, as it suggests that suboptimal intermediate models do not necessarily constrain the final achievable accuracy.
Improved Scaling Laws via Weak-to-Strong Generalization in Random Feature Ridge Regression
arXiv:2603.05691v3 Announce Type: replace Abstract: It is increasingly common in machine learning to use learned models to label data and then employ such data to train more capable models. The phenomenon of weak-to-strong generalization exemplifies the advantage of this two-stage procedure: a strong student is trained on imperfect labels obtained from a weak teacher, and yet the strong student outperforms the weak teacher. In this paper, we show that the potential improvement is substantial, in the sense that it affects the scaling law followed by the test error. Specifically, we consider students and teachers trained via random feature ridge regression (RFRR). Our main technical contribution is to derive a deterministic equivalent for the excess test error of the student trained on labels obtained via the teacher. Via this deterministic equivalent, we then identify regimes in which the scaling law of the student improves upon that of the teacher, unveiling that the improvement can be achieved both in bias-dominated and variance-dominated settings. Strikingly, the student may attain the minimax optimal rate regardless of the scaling law of the teacher -- in fact, when the test error of the teacher does not even decay with the sample size.
Abstract:It is increasingly common in machine learning to use learned models to label data and then employ such data to train more capable models. The phenomenon of weak-to-strong generalization exemplifies the advantage of this two-stage procedure: a strong student is trained on imperfect labels obtained from a weak teacher, and yet the strong student outperforms the weak teacher. In this paper, we show that the potential improvement is substantial, in the sense that it affects the scaling law followed by the test error. Specifically, we consider students and teachers trained via random feature ridge regression (RFRR). Our main technical contribution is to derive a deterministic equivalent for the excess test error of the student trained on labels obtained via the teacher. Via this deterministic equivalent, we then identify regimes in which the scaling law of the student improves upon that of the teacher, unveiling that the improvement can be achieved both in bias-dominated and variance-dominated settings. Strikingly, the student may attain the minimax optimal rate regardless of the scaling law of the teacher -- in fact, when the test error of the teacher does not even decay with the sample size.
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2603.05691 [cs.LG]
(or arXiv:2603.05691v3 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 205 cuvinte
Subiecte conexe
The Alignment Problem: Computer Science Meets Analytic Philosophy
LessWrong · 18 oct. 2026
Quadratic Weak-to-Strong Generalization in Random Feature Networks via Random Matrix Theory
arXiv · 8 oct. 2026
Game Design Creativity evals
LessWrong · 12 oct. 2026
Lattice-Based Cryptography Explained
Quantum Zeitgeist · 15 aug. 2026