analiza aiqbench
This research introduces CAP-SAM, a novel optimization technique that adapts perturbation radii based on local curvature to improve generalization in deep neural networks. By addressing the limitations of fixed-radius Sharpness-Aware Minimisation, the method achieves superior performance across diverse architectures and datasets while maintaining theoretical convergence guarantees. This advancement represents a significant step forward in optimizing loss landscape navigation for robust model training.
One Size Does Not Fit All: Curvature-Aware Adaptive Perturbation for SAM
arXiv:2610.20415v1 Announce Type: new Abstract: Poor generalisation in deep neural networks is frequently associated with convergence to sharp minima. Sharpness-Aware Minimisation (SAM) improves generalisation of deep neural networks by minimising a combination of the training loss and the local sharpness of the loss landscape, biasing convergence towards flat minima. However, SAM relies on a fixed perturbation radius $\rho$ that ignores local curvature, leading to inefficient behaviour: the perturbation may be too small to escape poor basins defined by sharp local minima, or too large to allow convergence to solutions with good loss-sharpness tradeoff. We reinterpret $\rho$ as the intensity of the perturbation signal and propose Curvature-Aware Adaptive Perturbation for SAM (CAP-SAM), a geometry-aware method that modulates the perturbation radius based on a sharpness ratio, enabling a more effective search for robust minima on the loss landscape. We prove that CAP-SAM preserves the same convergence rate as standard SAM in non-convex stochastic settings. Extensive experiments indicate that our approach consistently outperforms SAM and its most prominent variants across diverse network architectures on CIFAR-10, CIFAR-100, and ImageNet.
Abstract:Poor generalisation in deep neural networks is frequently associated with convergence to sharp minima. Sharpness-Aware Minimisation (SAM) improves generalisation of deep neural networks by minimising a combination of the training loss and the local sharpness of the loss landscape, biasing convergence towards flat minima. However, SAM relies on a fixed perturbation radius $\rho$ that ignores local curvature, leading to inefficient behaviour: the perturbation may be too small to escape poor basins defined by sharp local minima, or too large to allow convergence to solutions with good loss-sharpness tradeoff. We reinterpret $\rho$ as the intensity of the perturbation signal and propose Curvature-Aware Adaptive Perturbation for SAM (CAP-SAM), a geometry-aware method that modulates the perturbation radius based on a sharpness ratio, enabling a more effective search for robust minima on the loss landscape. We prove that CAP-SAM preserves the same convergence rate as standard SAM in non-convex stochastic settings. Extensive experiments indicate that our approach consistently outperforms SAM and its most prominent variants across diverse network architectures on CIFAR-10, CIFAR-100, and ImageNet.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.20415 [cs.LG]
(or arXiv:2610.20415v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.20415 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 267 cuvinte
Subiecte conexe
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization
arXiv · 5 sept. 2026
Curvature-Aware Radius Shrinkage for Adaptive Nearest Neighbor Classification
arXiv · 1 sept. 2026
TALAN: Task-Aligned Latent Adaptation Networks for Targeted Post-Training of Large Language Models
arXiv · 8 iun. 2026