aiqbench analysis
This paper establishes a rigorous mathematical framework linking parametric models to their native function spaces, unifying classical methods like kernel approaches with deep neural networks. It specifically demonstrates that the inductive bias of ReLU networks for depths greater than two resides within quasi-Banach spaces rather than standard normed spaces, offering new theoretical insights into model capacity and generalization limits.
Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks
arXiv:2606.14954v1 Announce Type: cross Abstract: We develop a general framework for analyzing representation costs of parametric data-fitting methods through their parameter-space regularizers. From this abstract perspective, we define representation costs for arbitrary parametric models and reveal their induced (native) function spaces. This unifies recent function-space views of data-fitting methods. We also prove that many natural results hold in this abstract setting, including representer theorems for parametric methods on their native spaces. The framework also rigorously connects parametric methods with their equivalent nonparametric descriptions under sufficient overparameterization. Classical methods and their native spaces, such as kernel methods / reproducing kernel Hilbert spaces, wavelets / Besov spaces, and shallow neural networks / variation spaces emerge as special cases of our abstract framework. A byproduct of "axiomatizing" the study of representation costs is that we also immediately obtain new results for deep neural networks: For depth-$L$ feedforward ReLU networks, their induced native spaces are $p$-normable quasi-Banach spaces with $p = 2/L$. This reveals that the inductive bias of deep neural networks (as given by the representation cost) cannot be captured by norms for depths $L > 2$.
Abstract:We develop a general framework for analyzing representation costs of parametric data-fitting methods through their parameter-space regularizers. From this abstract perspective, we define representation costs for arbitrary parametric models and reveal their induced (native) function spaces. This unifies recent function-space views of data-fitting methods. We also prove that many natural results hold in this abstract setting, including representer theorems for parametric methods on their native spaces. The framework also rigorously connects parametric methods with their equivalent nonparametric descriptions under sufficient overparameterization. Classical methods and their native spaces, such as kernel methods / reproducing kernel Hilbert spaces, wavelets / Besov spaces, and shallow neural networks / variation spaces emerge as special cases of our abstract framework. A byproduct of "axiomatizing" the study of representation costs is that we also immediately obtain new results for deep neural networks: For depth-$L$ feedforward ReLU networks, their induced native spaces are $p$-normable quasi-Banach spaces with $p = 2/L$. This reveals that the inductive bias of deep neural networks (as given by the representation cost) cannot be captured by norms for depths $L > 2$.
Functional Analysis (math.FA); Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)
arXiv:2606.14954 [math.FA]
(or arXiv:2606.14954v1 [math.FA] for this version)
https://doi.org/10.48550/arXiv.2606.14954 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 281 words
Related coverage
Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks
arXiv · Jun 17, 2026
Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks
arXiv · Jul 27, 2026
Amazon is investing in the Lean Focused Research Organization
Amazon Science · Jul 26, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026