aiqbench analysis
This paper introduces $\sigma$N-Ens, a novel implicit ensemble method that dynamically controls member diversity during training using softmax-temperature modulation to optimize the accuracy-calibration trade-off. By replicating only normalization layers, the approach achieves uncertainty estimation comparable to deep ensembles at significantly lower computational cost while supporting both convolutional and transformer architectures. The study provides theoretical insights into why this mechanism maintains calibration under input corruption but exhibits weaker out-of-distribution detection compared to explicit ensemble methods.
Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation
arXiv:2607.23860v1 Announce Type: new Abstract: Deep ensembles provide the most reliable uncertainty estimates in deep learning, but their cost grows linearly with the number of members. Implicit ensembles lower this cost by sharing a single backbone across members. Member diversity is a primary determinant of ensemble quality, yet no implicit ensemble can shape it during training; existing methods fix it at initialisation or build it into the architecture. We introduce $\sigma$N-Ens, a normalisation-based implicit ensemble that treats each member as a task in a multi-task architecture and modulates the shared backbone through sigmoid-bounded scalers. We also introduce a softmax-temperature regulariser, which shapes the equilibrium level of sharing between members and traces the accuracy-calibration frontier. Because only normalisation layers are replicated, the mechanism can wrap convolutional and transformer backbones alike, also allowing pretrained models to be adapted through a short fine-tune. We frame the epistemic uncertainty such an ensemble expresses as modulation uncertainty, and explain why its calibration holds under input corruption, and why its out-of-distribution detection is weaker. Our method is evaluated across ResNets and transformers on CIFAR-10/100, ImageNet and SST-2. $\sigma$N-Ens matches or outperforms deep ensembles at a fraction of their parameter cost, scales with ensemble size where partitioning methods collapse, and maintains calibration under distribution shift.
Abstract:Deep ensembles provide the most reliable uncertainty estimates in deep learning, but their cost grows linearly with the number of members. Implicit ensembles lower this cost by sharing a single backbone across members. Member diversity is a primary determinant of ensemble quality, yet no implicit ensemble can shape it during training; existing methods fix it at initialisation or build it into the architecture. We introduce $\sigma$N-Ens, a normalisation-based implicit ensemble that treats each member as a task in a multi-task architecture and modulates the shared backbone through sigmoid-bounded scalers. We also introduce a softmax-temperature regulariser, which shapes the equilibrium level of sharing between members and traces the accuracy-calibration frontier. Because only normalisation layers are replicated, the mechanism can wrap convolutional and transformer backbones alike, also allowing pretrained models to be adapted through a short fine-tune. We frame the epistemic uncertainty such an ensemble expresses as modulation uncertainty, and explain why its calibration holds under input corruption, and why its out-of-distribution detection is weaker. Our method is evaluated across ResNets and transformers on CIFAR-10/100, ImageNet and SST-2. $\sigma$N-Ens matches or outperforms deep ensembles at a fraction of their parameter cost, scales with ensemble size where partitioning methods collapse, and maintains calibration under distribution shift.
Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2607.23860 [cs.LG]
(or arXiv:2607.23860v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.23860 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 303 words
Related coverage
Phys: MIT Engineers Boost Coherence Times by 7× in Spin Ensembles
Quantum Zeitgeist · Jun 17, 2026
GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model
Meta Engineering · Aug 3, 2026
Debugging Logprob Mismatches in LLM Reinforcement Learning
AMD · Sep 23, 2026
Exploring self-distilled reasoning for supervised fine-tuning with Amazon Nova
AWS · Jul 21, 2026