aiqbench analysis
This paper addresses a critical limitation in the Muon optimizer by introducing Hilbert geometry to accurately bound probability distribution changes within softmax layers, where standard RMS metrics fail. By unifying recent optimization methods like RowNorm and Compositional Muon under this geometric framework, it establishes new parameter constraints for improving training stability in large language models and attention mechanisms. The work represents a significant theoretical advancement in optimizer design specifically tailored for the non-linear dynamics of modern deep learning architectures.
Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry
arXiv:2610.17588v1 Announce Type: cross Abstract: Muon controls weight updates through the RMS movement of a linear layer's output. This implicitly treats the linear output as a proxy for the full module, including the nonlinearity that follows. This proxy can be misleading for softmax, where the RMS shift in activations does not faithfully measure the change in the output distribution. We propose Hilbert distance as a natural metric for bounding changes in these output distributions. Hilbert distance bounds changes in cross-entropy loss and limits probability redistribution. For softmax, it takes the form of the variation seminorm of the preactivation perturbation, allowing probability constraints to be expressed as parameter constraints. Formulating the steepest descent problem with a Hilbert distance constraint brings two recently proposed methods under a unified mathematical framework. For the LM head, solving a row-wise approximation to the resulting diameter-constrained problem leads us to RowNorm \citep{pethick2025trainingdeeplearningmodels}. For scaled dot-product attention, the same principle yields the first-order query--key steepest descent problem underlying Compositional Muon \citep{keigwin2026compositional}. These connections motivate using Hilbert distance as a guiding principle for optimizer design in softmax modules.
Abstract:Muon controls weight updates through the RMS movement of a linear layer's output. This implicitly treats the linear output as a proxy for the full module, including the nonlinearity that follows. This proxy can be misleading for softmax, where the RMS shift in activations does not faithfully measure the change in the output distribution. We propose Hilbert distance as a natural metric for bounding changes in these output distributions. Hilbert distance bounds changes in cross-entropy loss and limits probability redistribution. For softmax, it takes the form of the variation seminorm of the preactivation perturbation, allowing probability constraints to be expressed as parameter constraints. Formulating the steepest descent problem with a Hilbert distance constraint brings two recently proposed methods under a unified mathematical framework. For the LM head, solving a row-wise approximation to the resulting diameter-constrained problem leads us to RowNorm \citep{pethick2025trainingdeeplearningmodels}. For scaled dot-product attention, the same principle yields the first-order query--key steepest descent problem underlying Compositional Muon \citep{keigwin2026compositional}. These connections motivate using Hilbert distance as a guiding principle for optimizer design in softmax modules.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.17588 [cs.LG]
(or arXiv:2610.17588v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.17588 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 272 words
Related coverage
Productionizing TurboQuant on AMD GPUs for KV-Cache-Bound LLM Inference
AMD · Jun 11, 2026
Okinawa Institute scientists unify three quantum reference frame methods
Quantum Zeitgeist · Aug 30, 2026
From CUDA to MLX: How K-Search Brings Decades of Kernel Expertise to Apple Silicon
BAIR Blog · Jul 29, 2026
Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective
arXiv · Jun 17, 2026