Statistical Properties of Deep Neural Networks with Dependent Data

Abstract:This paper develops theory for deep neural network (DNN) estimators under dependent data. To provide theory applicable to a variety of DNN-based estimators, I first establish nonasymptotic probability bounds on the theoretical and empirical $\mathcal{L}^{2}$-errors of nonparametric sieve estimators for a general class of estimation problems under possibly nonstationary $\beta$-mixing data taking values in unbounded sets. I then apply the theory to fully connected and convolutional DNN estimators without bounds or sparsity restrictions on the DNN weights. For both DNN classes, I derive general results when the function to be estimated is Hölder smooth and the data are nonstationary, subgaussian, and $\beta$-mixing with either exponential or polynomial decay. I then specialize these to nonparametric regression, logistic regression, and quantile regression settings. Under exponential $\beta$-mixing, the resulting estimators attain the nonparametric minimax rate of Stone (1982) up to logarithmic factors.

100 pages, 3 figures. V4 changes: Removed former Theorem 1 and associated alpha-mixing results; extended nonasymptotic beta-mixing bounds to nonstationary data; corrected measurability results and proofs. Added results for fixed-width architectures, convolutional architectures, polynomial mixing rates, and quantile regression. Partially linear model remains in arXiv:2410.22574

Machine Learning (stat.ML); Machine Learning (cs.LG); Econometrics (econ.EM)

arXiv:2410.11113 [stat.ML]

(or arXiv:2410.11113v4 [stat.ML] for this version)