aiqbench analysis
TukaBench addresses the critical gap in AI safety research by evaluating Large Language Models against culturally grounded jailbreak attempts across seven African languages, revealing that models are significantly more susceptible to prompt evasion when interacting with Low-Resource Languages than with English. The study highlights structural limitations in current evaluation frameworks, including model comprehension failures and the unreliability of automated LLM judges in these contexts, necessitating new metrics like "Deflection" to accurately capture safety nuances outside of English-centric datasets.
TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages
arXiv:2606.01322v2 Announce Type: replace-cross Abstract: Safety evaluation of Large Language Models (LLMs) remains heavily English-centric, leaving Low-Resource Languages (LRLs), particularly African ones, critically underexplored. We introduce TUKABENCH, a jailbreak benchmark for seven African languages that extends JailbreakBench (JBB) beyond direct translation through four settings: human translation of JBB prompts, English adaptation to African contexts followed by human translation, human-curated prompts validated through interactions with GPT-5.2, and code-switched prompts combining English and African languages, isolating the effect of language, cultural grounding, and prompt evasiveness on model safety. Across closed and open models, prompting in African languages reduces refusal relative to English, with culturally adapted prompts leading to least refusal. The evaluation also surfaces two structural limitations: model comprehension failures and reduced LLM-as-a-judge reliability in LRLs. To capture the first, we introduce Deflection alongside Refused and Jailbroken; to assess the second, we validate outputs with human annotations, showing that judge-human agreement drops in lower-resource languages and less commonly supported scripts.
Abstract:Safety evaluation of Large Language Models (LLMs) remains heavily English-centric, leaving Low-Resource Languages (LRLs), particularly African ones, critically underexplored. We introduce TUKABENCH, a jailbreak benchmark for seven African languages that extends JailbreakBench (JBB) beyond direct translation through four settings: human translation of JBB prompts, English adaptation to African contexts followed by human translation, human-curated prompts validated through interactions with GPT-5.2, and code-switched prompts combining English and African languages, isolating the effect of language, cultural grounding, and prompt evasiveness on model safety. Across closed and open models, prompting in African languages reduces refusal relative to English, with culturally adapted prompts leading to least refusal. The evaluation also surfaces two structural limitations: model comprehension failures and reduced LLM-as-a-judge reliability in LRLs. To capture the first, we introduce Deflection alongside Refused and Jailbroken; to assess the second, we validate outputs with human annotations, showing that judge-human agreement drops in lower-resource languages and less commonly supported scripts.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.01322 [cs.CL]
(or arXiv:2606.01322v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.01322 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 249 words
Related coverage
LLM Safety Alignment in Low-Resource Languages: A Systematic Literature Review
arXiv · Aug 19, 2026
Nemotron 3.5 Content Safety: Customizable Multimodal Safety for Global Enterprise AI
Hugging Face · Jun 4, 2026
KSAFE-MM: A Multimodal Safety Benchmark via Localized Contextualization for Korean Cultural Risks
arXiv · Oct 1, 2026
XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity
arXiv · Oct 15, 2026