aiqbench analysis
This research introduces $\mathrm{s}^{2}$HC, a novel architecture for hyper-connections that replaces unstable doubly stochastic constraints with spectral norm sphere confinement. By restoring control over the subdominant spectrum of residual matrices, the method prevents identity degeneration and eliminates parameterization inefficiencies like factorial scaling. This advancement offers a more stable and expressive framework for multi-stream model training without relying on iterative Sinkhorn-Knopp algorithms.
Spectral-Sphere-Constrained Hyper-Connections
arXiv:2603.20896v2 Announce Type: replace Abstract: Hyper-Connections (HC) extend residual connections into multiple streams, employing residual matrices for cross-stream mixing to enrich model expressivity. However, unconstrained mixing disrupts the identity mapping property intrinsic to the residual connection, causing unstable training. To address this, Manifold-Constrained Hyper-Connections (mHC) and its variants restrict these matrices to be doubly stochastic via Sinkhorn-Knopp (SK) algorithm or permutation-based parameterizations. We reveal three limitations of this doubly stochastic constraint: (1) identity degeneration, where learned matrices collapse around the identity initialization and diminish cross-stream interactions, (2) a expressivity bottleneck, where the doubly stochastic constraint restricts the freedom of the subdominant spectrum of the residual matrices, preventing the model from selectively preserving or attenuating cross-stream variations, and (3) parameterization inefficiencies, manifesting as unstable SK iterations or the factorial-scaling overhead of permutation-based parameterizations. To overcome these flaws, we propose Spectral-Sphere-Constrained Hyper-Connections ($\mathrm{s}^{2}$HC). By confining residual matrices to a spectral norm sphere, $\mathrm{s}^{2}$HC restores free control over the subdominant spectrum, enabling the model to selectively preserve or attenuate cross-stream variations. This shift eliminates unstable SK iterations and factorial parameterization, enabling expressive, non-degenerate residual matrices while preserving training stability.
Abstract:Hyper-Connections (HC) extend residual connections into multiple streams, employing residual matrices for cross-stream mixing to enrich model expressivity. However, unconstrained mixing disrupts the identity mapping property intrinsic to the residual connection, causing unstable training. To address this, Manifold-Constrained Hyper-Connections (mHC) and its variants restrict these matrices to be doubly stochastic via Sinkhorn-Knopp (SK) algorithm or permutation-based parameterizations. We reveal three limitations of this doubly stochastic constraint: (1) identity degeneration, where learned matrices collapse around the identity initialization and diminish cross-stream interactions, (2) a expressivity bottleneck, where the doubly stochastic constraint restricts the freedom of the subdominant spectrum of the residual matrices, preventing the model from selectively preserving or attenuating cross-stream variations, and (3) parameterization inefficiencies, manifesting as unstable SK iterations or the factorial-scaling overhead of permutation-based parameterizations. To overcome these flaws, we propose Spectral-Sphere-Constrained Hyper-Connections ($\mathrm{s}^{2}$HC). By confining residual matrices to a spectral norm sphere, $\mathrm{s}^{2}$HC restores free control over the subdominant spectrum, enabling the model to selectively preserve or attenuate cross-stream variations. This shift eliminates unstable SK iterations and factorial parameterization, enabling expressive, non-degenerate residual matrices while preserving training stability.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2603.20896 [cs.LG]
(or arXiv:2603.20896v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2603.20896 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 276 words
Related coverage
mHC-GNN: Manifold-Constrained Hyper-Connections for Graph Neural Networks
arXiv · Jul 21, 2026
Enabling DeepSeek-V4-Flash Training on AMD Instinct MI355X GPUs with Primus
AMD · Sep 3, 2026
Denoising Score Matching in Diffusion Models: The Effect of Rank Constraints on Generalization
arXiv · Oct 16, 2026
Kanno and Colleagues Propose Hodge Spectral Relaxations for Topology-Constrained Optimization
Quantum Zeitgeist · Jul 15, 2026