aiqbench analysis
The SDM activation function enhances standard softmax by incorporating similarity and distance metrics to improve robustness against covariate shifts and out-of-distribution inputs in selective classification tasks. By enabling interpretability through dense matching and data-driven partitioning of empirical CDFs, this approach offers a more reliable calibration method for pre-trained language models. This research provides a practical advancement for handling distributional changes in real-world deployment scenarios without sacrificing performance on in-distribution data.
Similarity-Distance-Magnitude Activations
arXiv:2509.12760v5 Announce Type: replace Abstract: We introduce the Similarity-Distance-Magnitude (SDM) activation function, a more robust and interpretable formulation of the standard softmax activation function, adding Similarity (i.e., correctly predicted depth-matches into training) awareness and Distance-to-training-distribution awareness to the existing output Magnitude (i.e., decision-boundary) awareness, and enabling interpretability-by-exemplar via dense matching. We further introduce the SDM estimator, based on a data-driven partitioning of the class-wise empirical CDFs via the SDM activation, to control the class- and prediction-conditional accuracy among selective classifications. When used as the final-layer activation over pre-trained language models for selective classification, the SDM estimator is more robust to covariate shifts and out-of-distribution inputs than existing calibration methods using softmax activations, while remaining informative over in-distribution data.
Abstract:We introduce the Similarity-Distance-Magnitude (SDM) activation function, a more robust and interpretable formulation of the standard softmax activation function, adding Similarity (i.e., correctly predicted depth-matches into training) awareness and Distance-to-training-distribution awareness to the existing output Magnitude (i.e., decision-boundary) awareness, and enabling interpretability-by-exemplar via dense matching. We further introduce the SDM estimator, based on a data-driven partitioning of the class-wise empirical CDFs via the SDM activation, to control the class- and prediction-conditional accuracy among selective classifications. When used as the final-layer activation over pre-trained language models for selective classification, the SDM estimator is more robust to covariate shifts and out-of-distribution inputs than existing calibration methods using softmax activations, while remaining informative over in-distribution data.
Accepted to Findings of the Association for Computational Linguistics: ACL 2026. 21 pages, 8 tables, 1 algorithm. arXiv admin note: substantial text overlap with arXiv:2502.20167
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2509.12760 [cs.LG]
(or arXiv:2509.12760v5 [cs.LG] for this version)
Full text under the publisher's open licence · 153 words
Related coverage
Manifold Steering from a Natural State is Effective and Generalizable, but Relies on Model Accuracy
LessWrong · Oct 12, 2026
Does Your LLM Trust You?
LessWrong · Aug 4, 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026
Confirming Claims of Superposition and Adversarial Examples in Toy Models
LessWrong · Aug 1, 2026