Conditional regression for the Nonlinear Single-Variable Model

Abstract:Regressing a function $F$ on $\mathbb{R}^d$ without incurring the statistical and computational curse of dimensionality requires exploitable structure. Compositional models $F=f\circ g$ in which $g$ has a low-dimensional range include classical single- and multi-index models as well as certain neural networks; while the case of linear $g$ is well understood, substantially less is known for nonlinear $g$. We study the model $F(X)=f(\Pi_\gamma X)$, where $\Pi_\gamma$ is the closest-point coordinate associated with an unknown regular curve $\gamma$, and $f$ is an unknown one-dimensional link function. The predictor $X$ need not be intrinsically low-dimensional and may have full-dimensional variation throughout a tubular neighborhood of the curve. We construct a nonparametric estimator based on response slicing, local principal component analysis, data-adaptive slice assignment, and one-dimensional local polynomial regression. Under coarse monotonicity of $f$ and sufficient variation normal to the curve relative to the observational noise and the coarse-monotonicity scale, the estimator attains, up to logarithmic factors, the minimax-optimal one-dimensional mean squared rate down to an explicit geometry- and noise-dependent saturation level. When the normal-variation condition is removed, we prove a complementary guarantee for the wide-slice regime. The estimator can be constructed in time $\mathcal{O}(d^2n\log n)$, and the constants and sample-size thresholds in our bounds depend at most polynomially on the ambient dimension $d$.

Machine Learning (stat.ML); Machine Learning (cs.LG)

arXiv:2411.09686 [stat.ML]

(or arXiv:2411.09686v4 [stat.ML] for this version)

https://doi.org/10.48550/arXiv.2411.09686 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.