aiqbench analysis
This paper introduces DiverValue-Bench, a novel benchmark designed to evaluate large language models across 74 countries by capturing fine-grained cultural and demographic variations often missed by aggregate metrics. The study demonstrates that lightweight fine-tuning techniques like LoRA and DPO can significantly improve value alignment within specific populations while maintaining robustness in out-of-domain scenarios. This work is critical for advancing equitable AI development and addressing the geographic disparities inherent in current model deployments.
DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Values
arXiv:2509.08022v3 Announce Type: replace Abstract: Aligning large language models (LLMs) with diverse human values is essential for safe and effective deployment, yet existing benchmarks often overlook cultural and demographic variation. We introduce DiverValue-Bench, a population-aware benchmark for evaluating multi-dimensional value alignment across 74 countries/regions. It contains 23,763 quality-controlled instances derived from PRISM user feedback and audited through large-scale human validation, with fine-grained value labels, personalized questions, contrastive reference answers, and rich demographic metadata. Using DiverValue-Bench, we evaluate representative LLMs and reveal substantial geographic and demographic disparities that are masked by aggregate performance. We further show that lightweight preference-based fine-tuning with Low-Rank Adaptation (LoRA) and Direct Preference Optimization (DPO) substantially improves in-domain value alignment while yielding consistent out-of-domain gains. These results highlight the need for population-aware alignment evaluation and demonstrate the utility of DiverValue-Bench as a practical foundation for global alignment, personalized value modeling, and equitable AI development.
Abstract:Aligning large language models (LLMs) with diverse human values is essential for safe and effective deployment, yet existing benchmarks often overlook cultural and demographic variation. We introduce DiverValue-Bench, a population-aware benchmark for evaluating multi-dimensional value alignment across 74 countries/regions. It contains 23,763 quality-controlled instances derived from PRISM user feedback and audited through large-scale human validation, with fine-grained value labels, personalized questions, contrastive reference answers, and rich demographic metadata. Using DiverValue-Bench, we evaluate representative LLMs and reveal substantial geographic and demographic disparities that are masked by aggregate performance. We further show that lightweight preference-based fine-tuning with Low-Rank Adaptation (LoRA) and Direct Preference Optimization (DPO) substantially improves in-domain value alignment while yielding consistent out-of-domain gains. These results highlight the need for population-aware alignment evaluation and demonstrate the utility of DiverValue-Bench as a practical foundation for global alignment, personalized value modeling, and equitable AI development.
11 pages, 5 figures. Accepted to IJCAI-ECAI 2026 (Human-Centred AI Special Track). v2: Updated to the camera-ready version
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2509.08022 [cs.CL]
(or arXiv:2509.08022v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2509.08022 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 256 words
Related coverage
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · Aug 3, 2026
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026
[AINews] not much happened today
Latent Space · Sep 18, 2026