Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks

Abstract:We develop a general framework for analyzing representation costs induced by parameter-space regularizers in data-fitting methods. For an arbitrary parametric method, we define its representation cost and native function space, prove existence, and identify conditions under which parameter-space and function-space problems have equal infimal values and minimizers transfer between them. This framework yields representer theorems and recovers classical formulations---including kernel methods and RKHSs, wavelets and Besov spaces, and shallow neural networks and variation spaces---as special cases. Our main new results concern depth-$L$ feedforward ReLU networks with weight-decay regularization. For these networks, we prove that the representation cost is a power of a quasi-seminorm and that, under suitable hypotheses, the native space is a quasi-Banach space with nonconvex unit ball when $L > 2$. These results identify a novel depth-dependent quasi-Banach function-space geometry induced by weight decay.

Functional Analysis (math.FA); Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)

arXiv:2606.14954 [math.FA]

(or arXiv:2606.14954v4 [math.FA] for this version)