aiqbench analysis
This study critically evaluates the muon optimizer by isolating its performance from confounding factors like network scale and architecture using low-rank matrix factorization as a controlled benchmark. The findings reveal that previously reported advantages over AdamW are not consistent in this setting, suggesting they may be artifacts of specific hyperparameter choices or complex model structures rather than inherent properties of the update rule itself. These results underscore the necessity for researchers to complement end-to-end benchmarks with rigorous evaluations on simplified, spectrally structured problems before claiming broad superiority for new optimization algorithms.
Reassessing Muon for Matrix Factorization
arXiv:2607.13246v2 Announce Type: replace-cross Abstract: Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training. Its empirical success has motivated a growing body of theoretical work that interprets Muon as steepest descent under the spectral norm. Yet it remains unclear which of Muon's advantages stem from its update rule itself and which are artifacts of the scale, architecture, and data of modern deep networks. In this work, we isolate the optimizer from these confounding factors by studying Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization. Through a controlled comparison against carefully tuned adaptive baselines, we find that Muon does not consistently outperform AdamW in this setting and that several previously reported advantages are sensitive to hyperparameter choices. Our results provide a more nuanced picture of when spectrum-aware orthogonalization is beneficial and argue for evaluating modern optimizers on controlled problems in addition to end-to-end benchmarks.
Abstract:Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training. Its empirical success has motivated a growing body of theoretical work that interprets Muon as steepest descent under the spectral norm. Yet it remains unclear which of Muon's advantages stem from its update rule itself and which are artifacts of the scale, architecture, and data of modern deep networks. In this work, we isolate the optimizer from these confounding factors by studying Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization. Through a controlled comparison against carefully tuned adaptive baselines, we find that Muon does not consistently outperform AdamW in this setting and that several previously reported advantages are sensitive to hyperparameter choices. Our results provide a more nuanced picture of when spectrum-aware orthogonalization is beneficial and argue for evaluating modern optimizers on controlled problems in addition to end-to-end benchmarks.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2607.13246 [cs.LG]
(or arXiv:2607.13246v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.13246 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 262 words
Related coverage
CacheMuon: Using Temporal Preconditioning To Approximate Polar Factor
arXiv · Jun 16, 2026
Optimizer Memory Schedules for Outscaling the Overtraining Axis
arXiv · Sep 7, 2026
Nonsmooth Optimization via Orthogonalized Momentum
arXiv · Sep 16, 2026
Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization
arXiv · Oct 6, 2026