How Does Preconditioning Guide Feature Learning in Deep Neural Networks?

Title:How Does Preconditioning Guide Feature Learning in Deep Neural Networks?

Abstract:Preconditioning is widely used in machine learning to accelerate convergence on the empirical risk, yet its role on the expected risk remains underexplored. In this work, we investigate how preconditioning affects feature learning and generalization performance. We first show that the input information available to the model is conveyed solely through the Gram matrix defined by the preconditioner's metric, thereby inducing a controllable spectral bias on feature learning. Concretely, instantiating the preconditioner as the $p$-th power of the input covariance matrix, we prove that, for a single ReLU neuron, increasing $p$ monotonically shifts the learned feature's sensitivity toward higher-variance eigenspaces, even though all choices of $p$ attain the same limiting population loss. We then empirically investigate how this spectral bias interacts with task-relevant features in three settings: robustness to noise, out-of-distribution generalization, and forward knowledge transfer. Our experiments show that learned representations favor the spectral components emphasized by preconditioning, and that generalization improves when this bias aligns with task-relevant features.

Machine Learning (cs.LG)

arXiv:2509.25637 [cs.LG]

(or arXiv:2509.25637v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2509.25637 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.