analiza aiqbench
This paper provides a rigorous theoretical analysis of Muon, an optimizer that leverages the matrix structure of neural network parameters to outperform standard gradient descent. By demonstrating how Muon exploits low-rank structures in Hessian matrices, this work bridges empirical success with formal convergence guarantees for advanced optimization techniques.
On the Convergence Analysis of Muon
arXiv:2505.23737v3 Announce Type: replace-cross Abstract: The majority of parameters in neural networks are naturally represented as matrices. However, most commonly used optimizers treat these matrix parameters as flattened vectors during optimization, potentially overlooking their inherent structural properties. Recently, an optimizer called Muon has been proposed, specifically designed to optimize matrix-structured parameters. Extensive empirical evidence shows that Muon can significantly outperform traditional optimizers when training neural networks. Nonetheless, the theoretical understanding of Muon's convergence behavior and the reasons behind its superior performance remain limited. In this work, we present a comprehensive convergence rate analysis of Muon and its comparison with Gradient Descent (GD). We characterize the conditions under which Muon can outperform GD. Our theoretical results reveal that Muon can benefit from the low-rank structure of Hessian matrices, a phenomenon widely observed in practical neural network training. Our experimental results support and corroborate the theoretical findings.
Abstract:The majority of parameters in neural networks are naturally represented as matrices. However, most commonly used optimizers treat these matrix parameters as flattened vectors during optimization, potentially overlooking their inherent structural properties. Recently, an optimizer called Muon has been proposed, specifically designed to optimize matrix-structured parameters. Extensive empirical evidence shows that Muon can significantly outperform traditional optimizers when training neural networks. Nonetheless, the theoretical understanding of Muon's convergence behavior and the reasons behind its superior performance remain limited. In this work, we present a comprehensive convergence rate analysis of Muon and its comparison with Gradient Descent (GD). We characterize the conditions under which Muon can outperform GD. Our theoretical results reveal that Muon can benefit from the low-rank structure of Hessian matrices, a phenomenon widely observed in practical neural network training. Our experimental results support and corroborate the theoretical findings.
Machine Learning (stat.ML); Information Theory (cs.IT); Machine Learning (cs.LG); Optimization and Control (math.OC)
arXiv:2505.23737 [stat.ML]
(or arXiv:2505.23737v3 [stat.ML] for this version)
Text integral sub licența deschisă a publicației · 161 cuvinte
Subiecte conexe
On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
arXiv · 6 aug. 2026
Convergence guarantees for Muon: New parameter regimes and generalizations
arXiv · 28 sept. 2026
Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
arXiv · 15 oct. 2026
Second-Moment Stochastic Approximation Methods
arXiv · 30 sept. 2026