analiza aiqbench
This research systematically evaluates intersectional fairness in six large language models, revealing a critical trade-off where no single model consistently achieves both reliability and bias mitigation across complex demographic combinations. The study highlights that while models may appear fair in ambiguous contexts, their performance diverges significantly when specific stereotypes are invoked, often favoring stereotype-reinforcing or counter-stereotype items depending on the attribute intersection. These findings underscore the urgent need for rigorous evaluation frameworks that assess subgroup fairness, accuracy, and response consistency simultaneously to prevent systemic bias in socially sensitive AI deployments.
Intersectional Fairness in Large Language Models
arXiv:2604.20677v3 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly deployed in socially sensitive settings, raising concerns about fairness and bias, particularly when multiple sensitive attributes intersect. We systematically evaluate intersectional fairness in six LLMs using two datasets from the Bias Benchmark for Question Answering (BBQ), combining race with gender and socioeconomic status. We assess bias, subgroup fairness, accuracy, and consistency across contexts and question polarities. While models perform well in ambiguous contexts, sparse non-unknown predictions limit fairness evaluation. In disambiguated contexts, stereotype alignment affects accuracy differently across datasets: models favor stereotype-reinforcing items in Race-SES but counter-stereotype items in Race-Gender for five of six models. Subgroup fairness metrics reveal uneven outcome distributions despite low observed disparities in some cases, while repeated runs show variable response consistency. Overall, no model consistently achieves both fairness and reliability, highlighting the need to evaluate bias, subgroup fairness, and consistency across intersectional contexts.
Abstract:Large Language Models (LLMs) are increasingly deployed in socially sensitive settings, raising concerns about fairness and bias, particularly when multiple sensitive attributes intersect. We systematically evaluate intersectional fairness in six LLMs using two datasets from the Bias Benchmark for Question Answering (BBQ), combining race with gender and socioeconomic status. We assess bias, subgroup fairness, accuracy, and consistency across contexts and question polarities. While models perform well in ambiguous contexts, sparse non-unknown predictions limit fairness evaluation. In disambiguated contexts, stereotype alignment affects accuracy differently across datasets: models favor stereotype-reinforcing items in Race-SES but counter-stereotype items in Race-Gender for five of six models. Subgroup fairness metrics reveal uneven outcome distributions despite low observed disparities in some cases, while repeated runs show variable response consistency. Overall, no model consistently achieves both fairness and reliability, highlighting the need to evaluate bias, subgroup fairness, and consistency across intersectional contexts.
Computation and Language (cs.CL)
arXiv:2604.20677 [cs.CL]
(or arXiv:2604.20677v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2604.20677 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 237 cuvinte
Subiecte conexe
Model-agnostic PII detection with LLMs
AWS · 10 sept. 2026
2026 BAIR Graduate Showcase
BAIR Blog · 1 iul. 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · 25 sept. 2026
Attribution Bias in Large Language Models
arXiv · 28 sept. 2026