analiza aiqbench
This research introduces Operator-level Adaptive Muon Orthogonalization (AMO), a refinement to the Muon optimizer that dynamically allocates computational resources based on per-matrix geometry rather than using a uniform schedule. By addressing heterogeneity in orthogonalization difficulty across different operator types and training stages, AMO achieves consistent performance gains over standard Muon and AdamW baselines on large-scale language models. The method demonstrates robust improvements in downstream tasks without requiring architectural changes, offering a practical optimization strategy for efficient pre-training.
AMO: Operator-level Adaptive Muon Orthogonalization
arXiv:2605.17806v2 Announce Type: replace Abstract: Muon has recently emerged as a competitive alternative to AdamW for large-scale pre-training, with orthogonalization via Newton-Schulz (NS) iteration as its core operation. Standard Muon applies a uniform NS schedule to all parameter matrices, overlooking possible differences in orthogonalization difficulty and its impact on performance. Through a systematic empirical study, we show that this per-matrix heterogeneity is pervasive and strongly associated with matrix geometry, which evolves dynamically across operator types, training stages, and network depths. Therefore, uniform NS schedules can lead to uneven orthogonalization quality across the model. Motivated by these findings, we propose Operator-level Adaptive Muon Orthogonalization (AMO), an observe-then-commit method that measures weight geometry by operator type early in training and then uses these signals to allocate the NS budget for the remainder of training. AMO delivers consistent improvements over uniform-schedule Muon across standard, prolonged, and continual pre-training, surpassing the strongest baseline by +0.76 on Llama3.1-1.4B and +0.51 on Qwen3-1.7B in average downstream performance of 12 evaluation tasks, with gains persisting at Llama3.1-4B scale.
Abstract:Muon has recently emerged as a competitive alternative to AdamW for large-scale pre-training, with orthogonalization via Newton-Schulz (NS) iteration as its core operation. Standard Muon applies a uniform NS schedule to all parameter matrices, overlooking possible differences in orthogonalization difficulty and its impact on performance. Through a systematic empirical study, we show that this per-matrix heterogeneity is pervasive and strongly associated with matrix geometry, which evolves dynamically across operator types, training stages, and network depths. Therefore, uniform NS schedules can lead to uneven orthogonalization quality across the model. Motivated by these findings, we propose Operator-level Adaptive Muon Orthogonalization (AMO), an observe-then-commit method that measures weight geometry by operator type early in training and then uses these signals to allocate the NS budget for the remainder of training. AMO delivers consistent improvements over uniform-schedule Muon across standard, prolonged, and continual pre-training, surpassing the strongest baseline by +0.76 on Llama3.1-1.4B and +0.51 on Qwen3-1.7B in average downstream performance of 12 evaluation tasks, with gains persisting at Llama3.1-4B scale.
Machine Learning (cs.LG)
arXiv:2605.17806 [cs.LG]
(or arXiv:2605.17806v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.17806 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 258 cuvinte
Subiecte conexe
OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality
arXiv · 9 iun. 2026
Reflections on unlearning and inoculation
LessWrong · 20 sept. 2026
SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
arXiv · 24 iul. 2026
Cost-free Spectral Estimation for Adaptive Newton--Schulz in Matrix Optimizers
arXiv · 30 sept. 2026