The Illusion of Improvement: Reject Inference Strategies in Credit Scoring

arXiv:2606.18479v3 Announce Type: replace Abstract: Reject inference methods are widely used to mitigate survival bias in credit scoring, yet their effectiveness remains poorly understood. We systematically evaluate several such methods and uncover a structural failure mode: in a natural retraining cycle, models whose accuracy improves while recall collapses create an illusion of improvement that leads practitioners to believe the system is getting better when, in fact, its rejection quality---the ability to correctly screen out defaulters---is deteriorating. We then propose a controlled exploration strategy that breaks the feedback loop without statistical assumptions: the lender deliberately approves a fraction of rejected applicants and observes their true outcomes. We show that accuracy and rejection quality respond in opposite directions to exploration: accuracy is lower under full exploration than under no exploration, while rejection quality is higher. A lender optimizing accuracy on the pool it has itself assembled finds no exploration rate that improves it and will therefore not explore, which is the worst choice for learning. Even minimal exploration rates (2\%--5\%) prove sufficient in our experiments to diagnose the severity of the feedback loop at near-zero cost. Our findings are consistent across two machine learning methods and three real-world datasets, and suggest that standard evaluation protocols are inadequate for assessing models trained under survival bias.

Abstract:Reject inference methods are widely used to mitigate survival bias in credit scoring, yet their effectiveness remains poorly understood. We systematically evaluate several such methods and uncover a structural failure mode: in a natural retraining cycle, models whose accuracy improves while recall collapses create an illusion of improvement that leads practitioners to believe the system is getting better when, in fact, its rejection quality---the ability to correctly screen out defaulters---is deteriorating. We then propose a controlled exploration strategy that breaks the feedback loop without statistical assumptions: the lender deliberately approves a fraction of rejected applicants and observes their true outcomes. We show that accuracy and rejection quality respond in opposite directions to exploration: accuracy is lower under full exploration than under no exploration, while rejection quality is higher. A lender optimizing accuracy on the pool it has itself assembled finds no exploration rate that improves it and will therefore not explore, which is the worst choice for learning. Even minimal exploration rates (2\%--5\%) prove sufficient in our experiments to diagnose the severity of the feedback loop at near-zero cost. Our findings are consistent across two machine learning methods and three real-world datasets, and suggest that standard evaluation protocols are inadequate for assessing models trained under survival bias.

Machine Learning (cs.LG); Computers and Society (cs.CY)

arXiv:2606.18479 [cs.LG]

(or arXiv:2606.18479v3 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2606.18479 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

The Illusion of Improvement: Reject Inference Strategies in Credit Scoring

Related coverage