aiqbench analysis
This research proposes using Stochastic Weight Averaging (SWA) as a computationally efficient alternative to large ensembles for enforcing symmetries in deep learning models trained with data augmentation. By modeling the training trajectory via an Ornstein–Uhlenbeck process, the authors demonstrate that SWA on augmented data yields equivariance benefits exceeding standard performance gains across computer vision and graph classification tasks. The study highlights a significant methodological advance in leveraging implicit regularization to reduce inference costs while maintaining theoretical symmetry properties.
Boosting Data Augmentation with Stochastic Weight Averaging
arXiv:2608.14373v1 Announce Type: new Abstract: The symmetries of a learning task have become an important factor in designing modern deep learning solutions. Data augmentation is a straightforward and effective way of incorporating symmetries into a generic neural network. Recent results show that infinitely large deep ensembles show perfect symmetry when trained on augmented data. However, since training ensembles requires repeating the training process many times, this method is costly. In this work, we study stochastic weight averaging (SWA) as an alternative ensembling technique that does not require repeated training runs. We analyze SWA by approximating the stochastic training trajectory at the end of training with an Ornstein--Uhlenbeck process. We show that in the infinite-width limit, SWA on augmented data provides an equiviariance boost that goes beyond what could be expected from the performance increase due to SWA alone. We verify our results with extensive numerical experiments on numerous models spanning computer vision and graph classification with both discrete and continuous symmetries.
Abstract:The symmetries of a learning task have become an important factor in designing modern deep learning solutions. Data augmentation is a straightforward and effective way of incorporating symmetries into a generic neural network. Recent results show that infinitely large deep ensembles show perfect symmetry when trained on augmented data. However, since training ensembles requires repeating the training process many times, this method is costly. In this work, we study stochastic weight averaging (SWA) as an alternative ensembling technique that does not require repeated training runs. We analyze SWA by approximating the stochastic training trajectory at the end of training with an Ornstein--Uhlenbeck process. We show that in the infinite-width limit, SWA on augmented data provides an equiviariance boost that goes beyond what could be expected from the performance increase due to SWA alone. We verify our results with extensive numerical experiments on numerous models spanning computer vision and graph classification with both discrete and continuous symmetries.
Machine Learning (cs.LG)
arXiv:2608.14373 [cs.LG]
(or arXiv:2608.14373v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.14373 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 250 words
Related coverage
Exploring self-distilled reasoning for supervised fine-tuning with Amazon Nova
AWS · Jul 21, 2026
Zero-Shot Heart Rate Variability Forecasting from Consumer Wearables Using Time Series Foundation Models
arXiv · Jul 23, 2026
Graph Data Augmentation via Contrastive Generator Inversion ($\texttt{DCBA}$)
arXiv · Oct 6, 2026
Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
METR · Jul 21, 2026