aiqbench analysis
This research advances optimization techniques for deep learning by integrating matrix-aware geometry into Sharpness-Aware Minimization (SAM) to enhance model generalization. By combining spectral inner perturbations with the Muon optimizer, the study achieves superior validation accuracy on standard benchmarks like ImageNet-1K compared to existing methods. The findings highlight a critical direction in machine learning: leveraging specific weight structures and geometric constraints to improve robustness without increasing computational overhead.
Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm
arXiv:2607.26001v1 Announce Type: new Abstract: Sharpness-Aware Minimization (SAM) aims to improve generalization by encouraging insensitivity to small, worst-case parameter perturbations. However, the notion of a "small" perturbation is inherently geometry-dependent: while existing SAM variants have explored a wide range of choices, a clear perspective on which geometries are most effective in practice remains elusive. Recent work on matrix-aware optimization, particularly the Muon optimizer, suggests that respecting the matrix structure of hidden-layer weights can lead to strong empirical performance. Motivated by this, we study matrix-aware geometry in both stages of SAM: we introduce a layerwise spectral inner perturbation for matrix-valued hidden-layer parameters and combine it with either AdamW/SGDW or Muon in the outer update. Across ImageNet-1K experiments on ViT-Small/16 and ResNet-50, we find that the combination of a spectral inner step with a Muon outer step performs consistently strongly, achieving the best validation accuracy on both models among the evaluated methods.
Abstract:Sharpness-Aware Minimization (SAM) aims to improve generalization by encouraging insensitivity to small, worst-case parameter perturbations. However, the notion of a "small" perturbation is inherently geometry-dependent: while existing SAM variants have explored a wide range of choices, a clear perspective on which geometries are most effective in practice remains elusive. Recent work on matrix-aware optimization, particularly the Muon optimizer, suggests that respecting the matrix structure of hidden-layer weights can lead to strong empirical performance. Motivated by this, we study matrix-aware geometry in both stages of SAM: we introduce a layerwise spectral inner perturbation for matrix-valued hidden-layer parameters and combine it with either AdamW/SGDW or Muon in the outer update. Across ImageNet-1K experiments on ViT-Small/16 and ResNet-50, we find that the combination of a spectral inner step with a Muon outer step performs consistently strongly, achieving the best validation accuracy on both models among the evaluated methods.
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2607.26001 [cs.LG]
(or arXiv:2607.26001v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.26001 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 242 words
Related coverage
AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
arXiv · Oct 2, 2026
Tensorion: A Tensor-Aware Generalization of the Muon Optimizer
arXiv · Aug 28, 2026
The Active Ingredient in Muon's Grokking
arXiv · Jul 24, 2026
Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimization
arXiv · Jul 23, 2026