aiqbench analysis
This paper introduces "verdict instability," a metric quantifying how post-hoc out-of-distribution (OOD) detection scores fluctuate when the reference set is resampled. The authors derive a closed-form expression for this instability, linking it directly to within-class dispersion and class counts, which proves particularly relevant under conditions of class imbalance. Their findings suggest that relying on distance-based scores for OOD queries can be misleading, as these often favor reproducible verdicts over those with higher local variance, potentially leading to suboptimal abstention strategies.
Verdict Instability of OOD Scores under Reference Resampling
arXiv:2609.00691v1 Announce Type: new Abstract: Post-hoc out-of-distribution detectors are fitted on a finite reference set, so every score they produce is an estimate. If we had chosen a different set, some verdicts would have moved. We measure that movement by resampling the reference set and recording the bootstrap standard deviation of the score, which we call verdict instability. It admits a closed form with no fitted parameters. The instability of a verdict is the within-class dispersion of the assigned class along the query's direction, divided by the square root of that class's reference count. That count is what separates verdict instability from the geometry of the score distribution, and it is identifiable only under class imbalance. Instability grows with the local dispersion. Far-OOD queries lie along the low-variance directions of an anisotropic embedding, so every distance-based score we test assigns its highest values to the verdicts that are most reproducible. Only estimators of local dispersion carry the sign a practitioner expects. We give a rule that predicts this sign for any score from a single label-free correlation, and abstention driven by a wrong-signed score turns out worse than abstention at random on every dataset we test.
Abstract:Post-hoc out-of-distribution detectors are fitted on a finite reference set, so every score they produce is an estimate. If we had chosen a different set, some verdicts would have moved. We measure that movement by resampling the reference set and recording the bootstrap standard deviation of the score, which we call verdict instability. It admits a closed form with no fitted parameters. The instability of a verdict is the within-class dispersion of the assigned class along the query's direction, divided by the square root of that class's reference count. That count is what separates verdict instability from the geometry of the score distribution, and it is identifiable only under class imbalance. Instability grows with the local dispersion. Far-OOD queries lie along the low-variance directions of an anisotropic embedding, so every distance-based score we test assigns its highest values to the verdicts that are most reproducible. Only estimators of local dispersion carry the sign a practitioner expects. We give a rule that predicts this sign for any score from a single label-free correlation, and abstention driven by a wrong-signed score turns out worse than abstention at random on every dataset we test.
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2609.00691 [cs.LG]
(or arXiv:2609.00691v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.00691 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 288 words
Related coverage
The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
arXiv · Jun 15, 2026
BiAxisBias: Evaluating LLM Bias Beyond a Single Prompt and a Single Explanation
arXiv · Aug 18, 2026
Europe’s leading AI model still can’t reliably follow European law
LessWrong · Oct 9, 2026
Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop
arXiv · Jul 28, 2026