FairNVT: Fair Classification via Noise Injection in Vision Transformers

Abstract:This paper presents FairNVT, a lightweight debiasing framework for pretrained transformer-based encoders that improves prediction fairness while preserving task performance. FairNVT is motivated by the intuition that reducing sensitive-attribute information in the representation used by the downstream classifier can facilitate fairer predictions. Our approach learns task-relevant and sensitive embeddings via lightweight adapters, applies calibrated Gaussian noise to the sensitive embedding, and fuses it with the task representation. Together with orthogonality constraints and fairness regularization, these components jointly reduce sensitive-attribute leakage in the learned embeddings and encourage fairer downstream predictions. Across three datasets spanning vision and language, FairNVT reduces sensitive-attribute attacker accuracy, improves fairness metrics such as demographic parity difference and equalized odds, and maintains competitive task performance.

Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

arXiv:2604.16780 [cs.CV]

(or arXiv:2604.16780v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2604.16780 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.