analiza aiqbench
This research reveals a critical gap in subtype robustness where models remain systematically overconfident despite accuracy drops when encountering unseen fine-grained categories within known classes. The study demonstrates that generic image corruption triggers confidence adjustments, whereas novel taxonomy shifts do not, indicating that calibration metrics are essential for evaluating true model reliability beyond simple accuracy scores. Consequently, the field must shift focus from solely monitoring accuracy to explicitly assessing calibration under distributional novelty in hierarchical data settings.
Subtype Robustness Is Not Just Accuracy: Calibration Under Unseen Subtype Shift
arXiv:2608.00928v1 Announce Type: new Abstract: Subtype robustness asks whether a model keeps the correct coarse prediction when test examples come from fine-grained subtypes absent from training but still inside a known coarse category. Prior work studies this almost entirely through accuracy. We ask whether the model also stays calibrated. We present the first systematic study of the question across ImageNet, BREEDS, iNaturalist and CIFAR-100 with five architectures. Calibration breaks down on unseen subtypes, where accuracy drops while confidence barely follows, leaving the model systematically overconfident exactly where it has become less accurate. At matched accuracy loss, generic image corruption causes a much larger drop in confidence, so the effect is not a general consequence of losing accuracy. The model reacts to visible degradation but not to in-taxonomy novelty. Recalibration tuned on seen subtypes narrows the gap but does not close it, and out-of-distribution scores flag the affected inputs only weakly. Subtype robustness should therefore be evaluated through calibration, not accuracy alone.
Abstract:Subtype robustness asks whether a model keeps the correct coarse prediction when test examples come from fine-grained subtypes absent from training but still inside a known coarse category. Prior work studies this almost entirely through accuracy. We ask whether the model also stays calibrated. We present the first systematic study of the question across ImageNet, BREEDS, iNaturalist and CIFAR-100 with five architectures. Calibration breaks down on unseen subtypes, where accuracy drops while confidence barely follows, leaving the model systematically overconfident exactly where it has become less accurate. At matched accuracy loss, generic image corruption causes a much larger drop in confidence, so the effect is not a general consequence of losing accuracy. The model reacts to visible degradation but not to in-taxonomy novelty. Recalibration tuned on seen subtypes narrows the gap but does not close it, and out-of-distribution scores flag the affected inputs only weakly. Subtype robustness should therefore be evaluated through calibration, not accuracy alone.
Machine Learning (cs.LG)
arXiv:2608.00928 [cs.LG]
(or arXiv:2608.00928v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.00928 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 250 cuvinte
Subiecte conexe
When 10,000 Windows Are Not 10,000 Tests: Auditing Statistical Confidence in Sliding-Window Time-Series Classification
arXiv · 28 sept. 2026
Dose-Aware Cold Diffusion with Physics Consistency for Generalizable Low-Dose CT Reconstruction
arXiv · 17 sept. 2026
Toward Uncertainty-Aware and Generalizable Neural Decoding for Quantum LDPC Codes
arXiv · 3 sept. 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · 13 oct. 2026