aiqbench analysis
This research proposes a geometry-aware initialization method for sigmoidal MLPs that encodes class-specific spectral data directly into network weights before training begins. By leveraging SVD to estimate principal directions and constructing "slab gates" from labeled sample geometry, the approach creates a more informative starting state than standard task-agnostic initializers like Xavier without sacrificing final accuracy. The findings suggest that these geometric features remain valuable even as fixed components, offering a potential pathway to reduce training overhead for specific image classification tasks.
From Dataset Spectral Geometry to Network Weights: A Geometry-Aware Initialization for Sigmoidal MLPs in Image Classification
arXiv:2606.28444v2 Announce Type: replace-cross Abstract: Classical universal approximation theorems (UAT) establish the expressive power of sigmoidal multilayer perceptrons, but they do not specify how the weights should be initialized. We study a supervised, data-dependent, geometry-aware initialization for one-hidden-layer sigmoidal MLPs that compiles labeled class geometry into network weights. The construction starts from the idea that sigmoid units can act as smooth half-space gates. For each class, we center the training samples at their mean, apply SVD to estimate principal directions and spectral scales, select retained directions by an energy threshold, and represent each retained direction by a pair of sigmoid slab gates. These class-specific gates are then concatenated into a shared hidden layer initialized directly from the training set. We also formulate a SVD-Mahalanobis subspace classifier as a non-neural geometric reference, which tests whether the estimated spectral class geometry is already discriminative before being embedded into the MLP. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 show that the proposed initializer produces a substantially more informative zero-epoch state than a matched task-agnostic Xavier reference, while full training reaches comparable final accuracy. Frozen-hidden experiments and neutral-head ablations further show that the class-wise SVD gates remain useful fixed features, even after the initially aligned output layer is replaced by a Xavier-initialized one.
Abstract:Classical universal approximation theorems (UAT) establish the expressive power of sigmoidal multilayer perceptrons, but they do not specify how the weights should be initialized. We study a supervised, data-dependent, geometry-aware initialization for one-hidden-layer sigmoidal MLPs that compiles labeled class geometry into network weights. The construction starts from the idea that sigmoid units can act as smooth half-space gates. For each class, we center the training samples at their mean, apply SVD to estimate principal directions and spectral scales, select retained directions by an energy threshold, and represent each retained direction by a pair of sigmoid slab gates. These class-specific gates are then concatenated into a shared hidden layer initialized directly from the training set. We also formulate a SVD-Mahalanobis subspace classifier as a non-neural geometric reference, which tests whether the estimated spectral class geometry is already discriminative before being embedded into the MLP. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 show that the proposed initializer produces a substantially more informative zero-epoch state than a matched task-agnostic Xavier reference, while full training reaches comparable final accuracy. Frozen-hidden experiments and neutral-head ablations further show that the class-wise SVD gates remain useful fixed features, even after the initially aligned output layer is replaced by a Xavier-initialized one.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
arXiv:2606.28444 [cs.LG]
(or arXiv:2606.28444v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.28444 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 301 words
Related coverage
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
Reflections on unlearning and inoculation
LessWrong · Sep 20, 2026
Kanno and Colleagues Propose Hodge Spectral Relaxations for Topology-Constrained Optimization
Quantum Zeitgeist · Jul 15, 2026
Joint Precision Neural Networks: Task-Aware Dependency and Predictive Learning
arXiv · Oct 6, 2026