analiza aiqbench
This research presents a model-agnostic framework that significantly improves the calibration of black-box radiology AI systems using test-time augmentation, addressing critical safety concerns in clinical decision-making. By applying geometric and physics-inspired perturbations without needing model internals, the proposed DualTTA method reduces expected calibration error more effectively than standard techniques like MC Dropout or Deep Ensembles. This approach offers a practical pathway to enhance the reliability of proprietary medical AI tools where access to internal parameters is restricted.
Improving Calibration of Black-Box Radiology AI Using Test-Time Augmentation
arXiv:2609.29931v1 Announce Type: new Abstract: Radiology AI systems increasingly inform clinical decisions such as triage, follow-up imaging, and treatment planning. For these decisions to be made safely, model outputs must be well calibrated, meaning predicted probabilities accurately reflect true risk. Many standard techniques for improving calibration, such as MC Dropout and Deep Ensembles, require access to model parameters or retraining. However, proprietary clinical AI systems operate as black boxes, preventing access to the model's internals. To that end, we propose a model-agnostic framework for improving calibration of black-box models using clinically grounded test-time augmentation (TTA). Our framework applies geometric and physics-inspired 3D CT perturbations and learns probability-level aggregation strategies without access to model internals or the original training data. Across pulmonary embolism and intracranial hemorrhage detection tasks, DualTTA achieved the strongest overall calibration among TTA methods, reducing the Expected Calibration Error by 54% (0.239 -> 0.109) and 43% (0.051 -> 0.029), respectively, while requiring only input-output access. Additionally, DualTTA outperformed uncertainty estimation techniques that require access to model internals, such as Temperature Scaling, MC Dropout, and Deep Ensembles, in most calibration metrics. These results demonstrate that learned TTA aggregation can improve the calibration of clinical AI systems, providing a practical approach for improving the reliability of black-box medical AI.
Abstract:Radiology AI systems increasingly inform clinical decisions such as triage, follow-up imaging, and treatment planning. For these decisions to be made safely, model outputs must be well calibrated, meaning predicted probabilities accurately reflect true risk. Many standard techniques for improving calibration, such as MC Dropout and Deep Ensembles, require access to model parameters or retraining. However, proprietary clinical AI systems operate as black boxes, preventing access to the model's internals. To that end, we propose a model-agnostic framework for improving calibration of black-box models using clinically grounded test-time augmentation (TTA). Our framework applies geometric and physics-inspired 3D CT perturbations and learns probability-level aggregation strategies without access to model internals or the original training data. Across pulmonary embolism and intracranial hemorrhage detection tasks, DualTTA achieved the strongest overall calibration among TTA methods, reducing the Expected Calibration Error by 54% (0.239 -> 0.109) and 43% (0.051 -> 0.029), respectively, while requiring only input-output access. Additionally, DualTTA outperformed uncertainty estimation techniques that require access to model internals, such as Temperature Scaling, MC Dropout, and Deep Ensembles, in most calibration metrics. These results demonstrate that learned TTA aggregation can improve the calibration of clinical AI systems, providing a practical approach for improving the reliability of black-box medical AI.
11 pages, 3 figures, 1 table. Accepted at the MICCAI 2026 Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging (UNSURE 2026)
Machine Learning (cs.LG); Human-Computer Interaction (cs.HC)
arXiv:2609.29931 [cs.LG]
(or arXiv:2609.29931v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.29931 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 250 cuvinte
Subiecte conexe
Problems of Proliferation
LessWrong · 29 sept. 2026
Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement
Microsoft Research · 11 aug. 2026
AI in healthcare: applications and best practices
Databricks · 24 iul. 2026
If This Were a Test, How Much Would It Cost?
LessWrong · 16 iun. 2026