analiza aiqbench
This research investigates the dynamics of manifold-constrained training in large language models, proposing a new optimizer called MACRO that eliminates weight decay by leveraging Riemannian optimization principles. The study demonstrates that these constraints effectively mitigate update cancellation and enhance robustness against low-precision computation while maintaining competitive performance with standard pre-training methods.
Demystifying Manifold Constraints in LLM Pre-training
arXiv:2605.04418v2 Announce Type: replace Abstract: The recent success of matrix optimizers (e.g., Muon) suggests that specific normalization of momentum, such as orthogonalization and row-wise normalization, benefits both the stability and acceleration of LLM training. Consequently, several recent studies have suggested that weights should also be normalized, leading to a Riemannian optimization problem. While such constrained training frameworks demonstrate superior performance, the effects of explicitly constraining weights, and their interaction with existing stabilization mechanisms, remain less understood. To bridge this gap, we study manifold constrained training dynamics through activation scales, rotational dynamics, and the update-to-weight ratio. We propose a Riemannian spectral steepest descent optimizer called MACRO, alongside a radius selection principle to serve as our testbed. Our analysis and numerical experiments reveal that RMSNorm and manifold constraints serve overlapping roles, and that weight decay can be completely eliminated when manifold constraints are applied. By controlling the update-to-weight ratio, constrained training significantly alleviates update cancellation, empirically demonstrating that MACRO is robust to low-precision computation and competitive with existing algorithms for standard LLM pre-training.
Abstract:The recent success of matrix optimizers (e.g., Muon) suggests that specific normalization of momentum, such as orthogonalization and row-wise normalization, benefits both the stability and acceleration of LLM training. Consequently, several recent studies have suggested that weights should also be normalized, leading to a Riemannian optimization problem. While such constrained training frameworks demonstrate superior performance, the effects of explicitly constraining weights, and their interaction with existing stabilization mechanisms, remain less understood. To bridge this gap, we study manifold constrained training dynamics through activation scales, rotational dynamics, and the update-to-weight ratio. We propose a Riemannian spectral steepest descent optimizer called MACRO, alongside a radius selection principle to serve as our testbed. Our analysis and numerical experiments reveal that RMSNorm and manifold constraints serve overlapping roles, and that weight decay can be completely eliminated when manifold constraints are applied. By controlling the update-to-weight ratio, constrained training significantly alleviates update cancellation, empirically demonstrating that MACRO is robust to low-precision computation and competitive with existing algorithms for standard LLM pre-training.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC)
arXiv:2605.04418 [cs.LG]
(or arXiv:2605.04418v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.04418 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 265 cuvinte
Subiecte conexe
ManifoldCache: Training-Free Diffusion Acceleration via Constraint Manifold Caching
arXiv · 7 oct. 2026
Exploring the Intrinsic Geometry of Diffusion Models with Constrained Inverse Kinematics
arXiv · 22 iul. 2026
Trajectory Planning without Trajectory Data: A Manifold-Guided Approach
arXiv · 8 oct. 2026
CALOS: Control-Affine Lyapunov On-manifold Safety Layer for Safe Deep Reinforcement Learning for Quadrotors
arXiv · 17 sept. 2026