An empirical evaluation of the risks of AI model updates using clinical data: stability, arbitrariness, and fairness

Abstract:Artificial Intelligence (AI) and Machine Learning (ML) models used in clinical settings are increasingly deployed to support clinical decision-making. However, when training data become stale due to changes in demographics, environment, or patient behaviors, model performance can degrade substantially. While updating models with new training data is necessary, such updates may also introduce new risks. We evaluated the proposed monitoring framework on four publicly available U.S.-based Type 1 Diabetes datasets containing high-resolution continuous glucose monitoring (CGM) data, comprising approximately 11,300 weekly observations from 496 participants younger than 20 years. All datasets included structured sociodemographic information. Using the prediction of severe hyperglycemia events in children with Type 1 Diabetes as a case study, we examine how different model update strategies can adversely affect model stability by causing predictions to change for a large number of cases after retraining, increase prediction arbitrariness, and worsen subgroup fairness and the balance of error rates across populations. We propose multiple dimensions for continuous monitoring to detect these issues and argue that such monitoring is essential for the development of trustworthy clinical decision support systems.

Best Paper Award in iEEE EMBC 2026. 4 pages, 3 figures

Artificial Intelligence (cs.AI)

arXiv:2604.23954 [cs.AI]

(or arXiv:2604.23954v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2604.23954 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.