aiqbench analysis
HalfNet demonstrates that predictive performance in neural networks is driven more by the geometry of weight space than by specific parameter values, allowing models to match standard architectures with significantly fewer parameters. By learning a low-rank factorization for random weight distributions, this approach effectively implements a data-dependent random embedding that bridges supervised metric learning and kernel perspectives. The method offers a promising direction for parameter-efficient models, suggesting that accuracy scales smoothly with the learned rank rather than relying on exhaustive training.
HalfNet: Randomized Neural Networks with Learned Subspace Geometry
arXiv:2606.04583v1 Announce Type: new Abstract: Many researchers investigated neural networks with some of their weights fixed to values randomly drawn from a given distribution, e.g., $N(0, I)$. Our proposed HalfNet draws random weights from $N(0, \Sigma)$, where $\Sigma$, which defines the geometry of the distribution, has a low-rank factorization that we learn from data. Experiments on MNIST and CIFAR-10 demonstrate that HalfNet can match the performance of fully trained multilayer perceptrons while using substantially fewer parameters. Spectral analysis indicates that much of the predictive power of neural networks lies in the geometry of their weight space rather than in the precise values of individual parameters, and we observe that accuracy scales smoothly with rank. HalfNet is not a neural architecture trick for low-rank structure; it implements a data-dependent random embedding that can also be interpreted through supervised metric learning, or random-feature and kernel perspectives.
Abstract:Many researchers investigated neural networks with some of their weights fixed to values randomly drawn from a given distribution, e.g., $N(0, I)$. Our proposed HalfNet draws random weights from $N(0, \Sigma)$, where $\Sigma$, which defines the geometry of the distribution, has a low-rank factorization that we learn from data. Experiments on MNIST and CIFAR-10 demonstrate that HalfNet can match the performance of fully trained multilayer perceptrons while using substantially fewer parameters. Spectral analysis indicates that much of the predictive power of neural networks lies in the geometry of their weight space rather than in the precise values of individual parameters, and we observe that accuracy scales smoothly with rank. HalfNet is not a neural architecture trick for low-rank structure; it implements a data-dependent random embedding that can also be interpreted through supervised metric learning, or random-feature and kernel perspectives.
Machine Learning (cs.LG)
arXiv:2606.04583 [cs.LG]
(or arXiv:2606.04583v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.04583 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 233 words
Related coverage
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
Introducing PIRAMID: Physics-Informed Research for Ambitious Mechanistic Interpretability
LessWrong · Jul 25, 2026
Across the Loss Landscape with Progressive Growth
arXiv · Aug 27, 2026
Gradient Concentration, Not Weight Saliency, Explains Representation-Level Class Unlearning
arXiv · Jul 24, 2026