Risk reversal for least squares estimators under nested convex constraints

Abstract:In constrained stochastic optimization, one expects that restricting the feasible set, provided it still contains the true parameter, should not increase the statistical risk of the corresponding projection estimator. We show that this intuition can fail, even in basic settings. We investigate this phenomenon in the Gaussian sequence model. Given a compact, convex set $\Theta \subseteq \mathbb{R}^d$, one observes \[ Y = \theta^\star + \sigma Z, \qquad Z \sim N(0, I_d), \] and seeks to estimate an unknown $\theta^\star \in \Theta$. Here, the maximum likelihood estimator over $\Theta$ coincides with the least squares estimator (LSE), given by the Euclidean projection of $Y$ onto $\Theta$. We construct an explicit example exhibiting \emph{risk reversal}: for sufficiently large noise, there exist nested compact convex sets $\Theta_S \subsetneq \Theta_L$ and $\theta^\star \in \Theta_S$ such that the LSE constrained to $\Theta_S$ has strictly larger squared-error risk than the LSE constrained to $\Theta_L$. Moreover, we demonstrate that risk reversal can persist at the level of worst-case risk. Finally, we show that the phenomenon is not specific to Gaussian noise or squared-error risk, extending our results beyond both settings. We clarify this phenomenon by contrasting noise regimes. In the vanishing-noise limit, the risk is governed at first order by the statistical dimension of the tangent cone, and risk reversal cannot occur at the leading $\sigma^2$ scale. In the diverging-noise regime, the risk instead depends on the global geometry of the constraint sets, and the embedding of $\Theta_S$ within $\Theta_L$ can reverse the risk ordering. These results reveal a previously unrecognized failure mode of the LSE. They demonstrate that in sufficiently noisy settings, tightening a constraint can paradoxically degrade statistical performance.

Statistics Theory (math.ST); Machine Learning (cs.LG); Optimization and Control (math.OC)

arXiv:2601.16041 [math.ST]

(or arXiv:2601.16041v2 [math.ST] for this version)

https://doi.org/10.48550/arXiv.2601.16041 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.