analiza aiqbench
This research proposes a semi-orthogonal weight initialization method optimized specifically for deep ReLU networks by solving optimization problems on the Stiefel manifold. The approach aims to address critical training challenges in large-depth architectures, such as dying ReLUs and gradient vanishing, while maintaining stable signal flow. Empirical results indicate improved performance across various datasets, particularly in few-shot learning scenarios where network depth is a limiting factor.
Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks
arXiv:2509.00362v2 Announce Type: replace Abstract: Stable and efficient training of ReLU networks with large depth is highly sensitive to weight initialization. Improper initialization can cause permanent neuron inactivation dying ReLU and exacerbate gradient instability as network depth increases. Methods such as He, Xavier, and orthogonal initialization preserve variance or promote approximate isometry. However, they do not necessarily regulate the pre-activation mean or control activation sparsity, and their effectiveness often diminishes in very deep architectures. This work introduces a semi-orthogonal initialization specifically optimized for ReLU by solving an optimization problem on the Stiefel manifold, thereby preserving scale and calibrating the pre-activation statistics from the outset. A family of closed-form solutions and an efficient sampling scheme are derived. Theoretical analysis at initialization shows that the proposed initialization mitigates dying ReLU, slows the decay of activation variance, and alleviates gradient vanishing, thereby stabilizing signal and gradient flow in deep architectures. Empirically, across MNIST, Fashion-MNIST, multiple tabular datasets, few-shot settings, and ReLU-family activations, the proposed initialization maintains stable performance as network depth increases, with particularly clear advantages in few-shot learning and large-depth settings.
Abstract:Stable and efficient training of ReLU networks with large depth is highly sensitive to weight initialization. Improper initialization can cause permanent neuron inactivation dying ReLU and exacerbate gradient instability as network depth increases. Methods such as He, Xavier, and orthogonal initialization preserve variance or promote approximate isometry. However, they do not necessarily regulate the pre-activation mean or control activation sparsity, and their effectiveness often diminishes in very deep architectures. This work introduces a semi-orthogonal initialization specifically optimized for ReLU by solving an optimization problem on the Stiefel manifold, thereby preserving scale and calibrating the pre-activation statistics from the outset. A family of closed-form solutions and an efficient sampling scheme are derived. Theoretical analysis at initialization shows that the proposed initialization mitigates dying ReLU, slows the decay of activation variance, and alleviates gradient vanishing, thereby stabilizing signal and gradient flow in deep architectures. Empirically, across MNIST, Fashion-MNIST, multiple tabular datasets, few-shot settings, and ReLU-family activations, the proposed initialization maintains stable performance as network depth increases, with particularly clear advantages in few-shot learning and large-depth settings.
Machine Learning (cs.LG)
arXiv:2509.00362 [cs.LG]
(or arXiv:2509.00362v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2509.00362 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 266 cuvinte
Subiecte conexe
Quantum Metrological Detection of the Unruh Effect with Optimized Accelerated Two-Level Atoms
arXiv · 21 iul. 2026
Manifold-Constrained Initial Noise Optimization for Efficient Generative Model Alignment
arXiv · 2 oct. 2026
Can Chemically Inspired Parameter Initialization Mitigate Barren Plateaus in Variational Quantum Eigensolvers?
arXiv · 22 sept. 2026
CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Exploration
arXiv · 9 sept. 2026