aiqbench analysis
This arXiv preprint audits gender, racial, and religious bias in six LLMs across English, Spanish, French, and German, using locally written Reddit posts rather than translated benchmarks. The authors are careful to say their results measure choice concentration, not stereotype endorsement or causal country effects, so the findings are descriptive. For readers, the main takeaway is that aggregate bias scores can hide category- and language-specific behaviour, which matters for anyone evaluating models before deploying them in education or hiring. The paper is an unreviewed author preprint, so the results should be treated as provisional.
From Language to Country to Valence: A Multilingual Audit of Stereotype Bias Across LLMs
arXiv:2611.02585v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly considered for education, recruitment, workplace decision support, and other consequential applications, making pre-deployment social-bias detection essential. Yet major benchmarks are English-centred or translated, focus mainly on negative stereotypes, and may miss linguistic and cultural variation. We investigate gender, racial, and religious bias using 12,000 locally written Reddit posts in English, Spanish, French, and German. Only search terms are translated, preserving local context. The dataset is balanced by language and category; English gender is additionally balanced by stereotype valence. Three open-weight and three proprietary API-based models are evaluated with a two-blank design that reduces grammatical-gender cues. Religion has the highest absolute choice imbalance in every model-language condition. Spanish is highest for gender, whereas German is highest for race and religion. The evaluated API models score below the open-weight models throughout. Negative English-gender items exceed positive items; Australian and Mexican contexts exceed their Canadian and Spanish counterparts. These descriptive differences measure choice concentration, not stereotype endorsement or causal country effects. The crossed patterns show how aggregate scores can conceal category- and context-specific behaviour, motivating closer inspection before deployment.
Abstract:Large language models (LLMs) are increasingly considered for education, recruitment, workplace decision support, and other consequential applications, making pre-deployment social-bias detection essential. Yet major benchmarks are English-centred or translated, focus mainly on negative stereotypes, and may miss linguistic and cultural variation. We investigate gender, racial, and religious bias using 12,000 locally written Reddit posts in English, Spanish, French, and German. Only search terms are translated, preserving local context. The dataset is balanced by language and category; English gender is additionally balanced by stereotype valence. Three open-weight and three proprietary API-based models are evaluated with a two-blank design that reduces grammatical-gender cues. Religion has the highest absolute choice imbalance in every model-language condition. Spanish is highest for gender, whereas German is highest for race and religion. The evaluated API models score below the open-weight models throughout. Negative English-gender items exceed positive items; Australian and Mexican contexts exceed their Canadian and Spanish counterparts. These descriptive differences measure choice concentration, not stereotype endorsement or causal country effects. The crossed patterns show how aggregate scores can conceal category- and context-specific behaviour, motivating closer inspection before deployment.
10 pages. Author preprint; conference review ongoing
Computation and Language (cs.CL)
arXiv:2611.02585 [cs.CL]
(or arXiv:2611.02585v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2611.02585 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 284 words
Related coverage
Tracing Stereotypes from Representation to Output in Multilingual LLMs
arXiv · Sep 9, 2026
AmchiBias: Measuring Stereotypical Bias in Goan Identity Groups with a Minimal Pair Dataset in English and Konkani
arXiv · Jun 17, 2026
QuantiBias: Benchmarking Quantization-Induced Bias in LLMs
arXiv · Jul 25, 2026
Lost in Translation: Why Lower Bias Scores Don't Mean Fairer Models
arXiv · Oct 15, 2026