aiqbench analysis
This research advances self-distillation techniques by establishing theoretical guarantees for prediction-only regimes where labeled training data are unavailable. By deriving deterministic equivalents for ridge regression under proportional asymptotics, the study demonstrates that optimally mixed predictions can strictly outperform original teachers even with out-of-distribution covariates. The work also provides a practical method to estimate optimal mixing weights using minimal calibration sets without additional model fitting.
Prediction-Only Distillation in Linear and Logistic Regression
arXiv:2607.15450v1 Announce Type: cross Abstract: Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs. In many practical deployments, however, the labeled training data are no longer available, and one has access only to the trained predictor and fresh unlabeled covariates. We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme: a pure-distilled student is trained on fresh covariates pseudo-labeled by the teacher, and the final predictor is an affine combination of the teacher and student predictions. For ridge regression under proportional asymptotics, we derive deterministic equivalents for the optimally mixed prediction risk under general anisotropic covariance and deterministic signal. We show that this risk is strictly smaller than the teacher risk for almost every pair of teacher and student regularization levels, including when the fresh covariates are out-of-distribution and even when their covariance is isotropic. We further show that the optimal mixing weight cannot be identified from unlabeled data alone, but can be consistently estimated in a single post-training step using a small independent labeled calibration set, without additional model fitting. Finally, for binary logistic regression, we show that prediction mixing can outperform both the teacher and the pure-distilled classifier.
Abstract:Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs. In many practical deployments, however, the labeled training data are no longer available, and one has access only to the trained predictor and fresh unlabeled covariates. We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme: a pure-distilled student is trained on fresh covariates pseudo-labeled by the teacher, and the final predictor is an affine combination of the teacher and student predictions. For ridge regression under proportional asymptotics, we derive deterministic equivalents for the optimally mixed prediction risk under general anisotropic covariance and deterministic signal. We show that this risk is strictly smaller than the teacher risk for almost every pair of teacher and student regularization levels, including when the fresh covariates are out-of-distribution and even when their covariance is isotropic. We further show that the optimal mixing weight cannot be identified from unlabeled data alone, but can be consistently estimated in a single post-training step using a small independent labeled calibration set, without additional model fitting. Finally, for binary logistic regression, we show that prediction mixing can outperform both the teacher and the pure-distilled classifier.
Statistics Theory (math.ST); Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2607.15450 [math.ST]
(or arXiv:2607.15450v1 [math.ST] for this version)
https://doi.org/10.48550/arXiv.2607.15450 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 294 words
Related coverage
Hard labels sampled from sparse targets mislead rotation invariant algorithms
arXiv · Jun 8, 2026
REALM: Regime-Switching, Explainable, and Activation-Induced Linear Models
arXiv · Sep 30, 2026
7 Machine Learning Algorithms That Still Matter
KDnuggets · Jul 30, 2026
Interpretable AI with Local Distillation
arXiv · Aug 25, 2026