aiqbench analysis
BabelSafe and BabelGuard address a critical gap in LLM deployment by shifting from generic risk taxonomies to benchmarks grounded in specific regional regulations and cultural contexts. The proposed dLLM-based guardrail models offer scalable solutions for policy-conditioned safety assessment across 13 languages, moving beyond machine-translated data to ensure compliance with jurisdiction-specific requirements. This work provides a necessary framework for evaluating how large models handle diverse legal and cultural constraints in cross-linguistic applications.
BabelSafe: A Policy-Grounded Multilingual Safety Benchmark for LLMs
arXiv:2605.00689v2 Announce Type: replace Abstract: As Large Language Models (LLMs) are increasingly deployed in cross-linguistic contexts, ensuring safety across diverse regulatory and cultural environments has become a critical challenge. However, existing multilingual benchmarks largely rely on general risk taxonomies and machine-translated data, limiting evaluation to predefined risk categories and providing insufficient coverage of region-specific regulatory requirements and cultural contexts. To bridge these gaps, we introduce BabelSafe, a policy-grounded multilingual safety benchmark covering 13 language settings. BabelSafe is constructed from regional regulatory sources, with risk categories and fine-grained rules extracted from jurisdiction-specific regulatory documents directly used to guide the generation of multilingual safety data. During data generation, we further incorporate region-specific cultural contexts, enabling regulation-grounded and culturally contextualized evaluation across languages. Building on BabelSafe, we develop BabelGuard, a Diffusion Large Language Model (dLLM)-based guardrail model that supports multilingual safety judgment and policy-conditioned safety assessment. BabelGuard has two variants, a lightweight 1.5B model for fast `safe/unsafe' classification and a more capable 7B model for customizable policy-conditioned safety checking with detailed explanations. We evaluate BabelGuard against 11 strong guardrail baselines on 6 existing multilingual safety benchmarks and BabelSafe, demonstrating the strong performance of BabelGuard across these evaluation settings. We hope that BabelSafe and BabelGuard can help advance the development of regulation-aware and culturally contextualized multilingual guardrail systems.
Abstract:As Large Language Models (LLMs) are increasingly deployed in cross-linguistic contexts, ensuring safety across diverse regulatory and cultural environments has become a critical challenge. However, existing multilingual benchmarks largely rely on general risk taxonomies and machine-translated data, limiting evaluation to predefined risk categories and providing insufficient coverage of region-specific regulatory requirements and cultural contexts. To bridge these gaps, we introduce BabelSafe, a policy-grounded multilingual safety benchmark covering 13 language settings. BabelSafe is constructed from regional regulatory sources, with risk categories and fine-grained rules extracted from jurisdiction-specific regulatory documents directly used to guide the generation of multilingual safety data. During data generation, we further incorporate region-specific cultural contexts, enabling regulation-grounded and culturally contextualized evaluation across languages. Building on BabelSafe, we develop BabelGuard, a Diffusion Large Language Model (dLLM)-based guardrail model that supports multilingual safety judgment and policy-conditioned safety assessment. BabelGuard has two variants, a lightweight 1.5B model for fast `safe/unsafe' classification and a more capable 7B model for customizable policy-conditioned safety checking with detailed explanations. We evaluate BabelGuard against 11 strong guardrail baselines on 6 existing multilingual safety benchmarks and BabelSafe, demonstrating the strong performance of BabelGuard across these evaluation settings. We hope that BabelSafe and BabelGuard can help advance the development of regulation-aware and culturally contextualized multilingual guardrail systems.
Computation and Language (cs.CL); Cryptography and Security (cs.CR)
arXiv:2605.00689 [cs.CL]
(or arXiv:2605.00689v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.00689 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 306 words
Related coverage
NVIDIA GTC Taipei at COMPUTEX: Live Updates on What’s Next in AI
NVIDIA · May 21, 2026
Granite 4.1 LLMs: How They’re Built
Hugging Face · Apr 29, 2026
PSEBench: A Controllable and Verifiable Benchmark for Evaluating LLMs in Patient Safety Event Triage
arXiv · Jun 10, 2026
PSEBench: A Controllable and Verifiable Benchmark for Evaluating LLMs in Patient Safety Event Triage
arXiv · Jun 6, 2026