aiqbench analysis
This research exposes a critical vulnerability in current LLM safety alignment, revealing that filters trained primarily on English data often fail to prevent bias propagation in non-English languages like Bengali and Hinglish. The study introduces the INCLUDE benchmark to quantify these cross-lingual gaps across six Indian languages, highlighting how closed-source models can paradoxically exhibit higher bias than open-source counterparts in specific linguistic contexts. These findings underscore the urgent need for multilingual safety training protocols before deploying voice assistants and dialogue systems in diverse global markets.
Safety Alignment Illusion: The Cross-Lingual Safety Gap in LLMs
arXiv:2608.18131v1 Announce Type: new Abstract: Current safety alignment training for Large Language Models (LLMs) are heavily English-centric. When such safety filters fail for non-English languages, the consequences are immediate and user-facing: voice assistants and spoken dialogue systems may produce stereotype-reinforcing outputs, bypassing the standard English-focused safety alignments and propagating harmful bias to non-English speaking communities. For spoken language technologies deployed across India's linguistically diverse population, this represents a critical failure mode. To address this cross-lingual gap, we introduce INCLUDE (Indian Cultural Lens for Understanding and Detecting Embedded Biases), a multilingual evaluation benchmark designed to quantify Indian-centric socio-cultural biases. INCLUDE consists of 2,604 prompts spanning six prompt languages: English, Hindi, Bengali, Marathi, Tamil, and Hinglish (Hindi-English code-mix). We evaluate ten open- and closed-source LLMs against this benchmark, analyzing 14,988 bias scores. Our statistical results reveal two key findings. First, Bengali yielded the highest average bias score in open-source models. Second, English demonstrated a notable reversal, producing the lowest bias in open-source models but the highest bias in closed-source models.
Abstract:Current safety alignment training for Large Language Models (LLMs) are heavily English-centric. When such safety filters fail for non-English languages, the consequences are immediate and user-facing: voice assistants and spoken dialogue systems may produce stereotype-reinforcing outputs, bypassing the standard English-focused safety alignments and propagating harmful bias to non-English speaking communities. For spoken language technologies deployed across India's linguistically diverse population, this represents a critical failure mode. To address this cross-lingual gap, we introduce INCLUDE (Indian Cultural Lens for Understanding and Detecting Embedded Biases), a multilingual evaluation benchmark designed to quantify Indian-centric socio-cultural biases. INCLUDE consists of 2,604 prompts spanning six prompt languages: English, Hindi, Bengali, Marathi, Tamil, and Hinglish (Hindi-English code-mix). We evaluate ten open- and closed-source LLMs against this benchmark, analyzing 14,988 bias scores. Our statistical results reveal two key findings. First, Bengali yielded the highest average bias score in open-source models. Second, English demonstrated a notable reversal, producing the lowest bias in open-source models but the highest bias in closed-source models.
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY)
arXiv:2608.18131 [cs.AI]
(or arXiv:2608.18131v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.18131 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 265 words
Related coverage
LLM Safety Alignment in Low-Resource Languages: A Systematic Literature Review
arXiv · Aug 19, 2026
actr: aligning thoughts and responses for multilingual safety in reasoning llms
arXiv · Sep 30, 2026
Who Bridges Safety? Identifying and Targeting Cross-Lingual Shared Safety Pathways
arXiv · Oct 1, 2026
Multilingual Refusal Alignment for Safer Large Language Models
arXiv · Jun 9, 2026