aiqbench analysis
This research challenges the prevailing assumption that removing demographic information from model representations is sufficient for algorithmic fairness, demonstrating that such invariance can inadvertently degrade performance or generate new biases. The study provides a rigorous mathematical and empirical analysis showing that some encoding of demographics remains necessary when these factors correlate with target labels to ensure accurate predictions. Consequently, developers must move beyond simple representation erasure and adopt more nuanced strategies that balance fairness constraints with predictive utility.
Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias
arXiv:2609.32004v1 Announce Type: new Abstract: Encoded demographic information in internal model representations is a commonly assumed risk factor for algorithmic bias, with demographic representation invariance often being touted as the ideal state. However, while demographic shortcut learning is a genuine threat, some degree of encoding is necessary when demographics correlate with target labels. Here, we show, mathematically and empirically, that enforcing demographic invariance can actually hamper bias mitigation and even create new biases. We distinguish marginal from class-conditional representation invariance, and show that they imply the standard group fairness notions of demographic parity and equalized odds, respectively. We evaluate the effects on predictive performance and fairness of enforcing both invariance types, both theoretically and empirically across five tabular and two chest X-ray imaging datasets. Our findings support our mathematical argument that demographic representation invariance is neither desirable nor sufficient for fairness.
Abstract:Encoded demographic information in internal model representations is a commonly assumed risk factor for algorithmic bias, with demographic representation invariance often being touted as the ideal state. However, while demographic shortcut learning is a genuine threat, some degree of encoding is necessary when demographics correlate with target labels. Here, we show, mathematically and empirically, that enforcing demographic invariance can actually hamper bias mitigation and even create new biases. We distinguish marginal from class-conditional representation invariance, and show that they imply the standard group fairness notions of demographic parity and equalized odds, respectively. We evaluate the effects on predictive performance and fairness of enforcing both invariance types, both theoretically and empirically across five tabular and two chest X-ray imaging datasets. Our findings support our mathematical argument that demographic representation invariance is neither desirable nor sufficient for fairness.
An earlier version is available as a non-peer reviewed preprint, cited in the paper. This submission supersedes it with consent of all authors; all experiments are new
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2609.32004 [cs.LG]
(or arXiv:2609.32004v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.32004 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 184 words
