A Nonlinear Singular Value Theory for Neural Networks

Abstract:Recently Brown et al. [2025] established a singular value decomposition (SVD) for maps (especially nonlinear) satisfying certain norm conditions. We prove that most modern neural architectures admit this nonlinear SVD (NLSVD) representation---with no change in input--output behavior---and enumerate the classes covered. In this factorization the network is a left-invertible nonlinear map followed by a final linear layer. Moreover, the left-invertible factor is norm-preserving, so distances in the embedding (activations before the final linear layer) calibrate directly to distances in input space. We introduce a flexible architecture that yields an explicit decomposition at training time, a data-driven algorithm for estimating the representation from trained models, and the mathematical foundations for nonlinear analogues of row and null spaces in neural networks. Empirical case studies illustrate uses of the theory for latent-space pullback (visualization and data generation), bias detection, and membership-inference robustness under training. Altogether, these foundations support new approaches to core problems in neural-network analysis.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2605.06938 [cs.LG]

(or arXiv:2605.06938v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.06938 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.