Minimax Lower Bound for Estimating Diffusion-based Local Intrinsic Dimension

Abstract:While diffusion-based methods have recently emerged as effective tools for probing the intrinsic geometry of high-dimensional data, their statistical difficulty remains largely unexplored. We study estimation of the finite-scale population functional underlying FLIPD (Kamkari et al., 2024; arXiv:2406.03537), a diffusion-based local intrinsic dimension (LID) quantity defined through the logarithmic scale derivative of a Gaussian-smoothed density. Intuitively, Gaussian smoothing turns local dimension into a scale law: near a $d$-dimensional manifold, the kernel mass grows like $\sigma^d$, so differentiating with respect to the noise scale reveals the intrinsic exponent. Under a regular manifold model, we show uniformly over the model class that the finite-scale field differs from the manifold dimension $d$ by at most $O(\sigma^2)$. We then establish a minimax lower bound of order $(n\sigma^d)^{-1}$ for estimating this finite-scale field from $n$ observations, for $n^{-1/(2\alpha+d)}\lesssim\sigma\le\sigma_0$. At the smallest scale covered by our lower-bound construction, the bound becomes the nonparametric rate $n^{-2\alpha/(2\alpha+d)}$.

Machine Learning (stat.ML); Machine Learning (cs.LG); Statistics Theory (math.ST)

arXiv:2609.04822 [stat.ML]

(or arXiv:2609.04822v1 [stat.ML] for this version)

https://doi.org/10.48550/arXiv.2609.04822 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.