aiqbench analysis
This paper is primarily about machine learning theory: it extends covariate-shift severity analysis from single models to ensembles of random partition-based estimators and proves minimax-optimal convergence rates given sufficient ensemble size. The practical takeaway is that ensembles' robustness to distribution shift has a theoretical basis, and that more severe shifts plausibly require larger ensembles. As a theory result on partition-based kernel estimators, its relevance to large modern models is indirect and would need empirical validation.
Ensembles of Random Partition-Based Models under Covariate Shift
arXiv:2611.05195v1 Announce Type: new Abstract: Distribution shifts, including covariate shifts, are ubiquitous in real-world applications of machine learning. Empirical evidence suggests that, across a variety of models with and without downstream adaptation, ensembles tend to be more robust to covariate shift than single models. We extend the theory of covariate shift severity from single models to ensembles, providing theoretical justification for the empirical trends. We introduce new discrepancy measures of covariate shift severity to control the risk of partition-based kernel estimators. We then show that with sufficient ensemble size, these models can achieve minimax-optimal risk convergence rates under covariate shift. Our results suggest that modelers can expect to need larger ensemble sizes for more severe covariate shifts.
Abstract:Distribution shifts, including covariate shifts, are ubiquitous in real-world applications of machine learning. Empirical evidence suggests that, across a variety of models with and without downstream adaptation, ensembles tend to be more robust to covariate shift than single models. We extend the theory of covariate shift severity from single models to ensembles, providing theoretical justification for the empirical trends. We introduce new discrepancy measures of covariate shift severity to control the risk of partition-based kernel estimators. We then show that with sufficient ensemble size, these models can achieve minimax-optimal risk convergence rates under covariate shift. Our results suggest that modelers can expect to need larger ensemble sizes for more severe covariate shifts.
Machine Learning (cs.LG); Statistics Theory (math.ST)
arXiv:2611.05195 [cs.LG]
(or arXiv:2611.05195v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2611.05195 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 209 words