analiza aiqbench
D2VBench introduces a novel benchmark addressing critical gaps in evaluating large language models' value alignment within complex, real-world daily dilemmas. By combining 10,000 human-LLM co-created scenarios with fine-grained ethical concepts, this tool offers researchers a robust method to assess how well AI systems navigate conflicting moral priorities beyond simplistic test suites.
D2VBench: Benchmarking Large Language Models with Value Dilemmas in Daily Scenarios
arXiv:2607.19834v1 Announce Type: new Abstract: With the wide application of large language models (LLMs) in real-world scenarios, the value implication of their outputs is crucial. However, existing evaluation benchmarks suffer from insufficient coverage of value dilemmas in daily scenarios involving multiple value conflicts and simplistic evaluation formalisms that fail to assess LLMs' value alignment. To address these issues, we propose D2VBench, a value alignment benchmark comprising 10,000 instances of real daily dilemma scenarios constructed through a multi-stage collaboration between LLMs and humans, grounded in 158 manually annotated fine-grained value concepts. For evaluation on the benchmark, we present a hybrid evaluation paradigm that integrates multiple-choice questions with open-ended questions. We conduct comprehensive evaluations on eight mainstream LLMs. Experimental results demonstrate that D2VBench exhibits high reliability and robustness, effectively reflecting the LLMs' alignment across different value categories and dimensions, and providing a more realistic and fine-grained tool for research on value alignment. The dataset is available at https://github.com/tjunlp-lab/D2VBench.
Abstract:With the wide application of large language models (LLMs) in real-world scenarios, the value implication of their outputs is crucial. However, existing evaluation benchmarks suffer from insufficient coverage of value dilemmas in daily scenarios involving multiple value conflicts and simplistic evaluation formalisms that fail to assess LLMs' value alignment. To address these issues, we propose D2VBench, a value alignment benchmark comprising 10,000 instances of real daily dilemma scenarios constructed through a multi-stage collaboration between LLMs and humans, grounded in 158 manually annotated fine-grained value concepts. For evaluation on the benchmark, we present a hybrid evaluation paradigm that integrates multiple-choice questions with open-ended questions. We conduct comprehensive evaluations on eight mainstream LLMs. Experimental results demonstrate that D2VBench exhibits high reliability and robustness, effectively reflecting the LLMs' alignment across different value categories and dimensions, and providing a more realistic and fine-grained tool for research on value alignment. The dataset is available at this https URL.
Computation and Language (cs.CL)
arXiv:2607.19834 [cs.CL]
(or arXiv:2607.19834v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2607.19834 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 248 cuvinte
Subiecte conexe
Always ask your agent to be honest
LessWrong · 16 sept. 2026
Would a Large Language Model Pay Extra for a View? Inferring Willingness to Pay from Subjective Choices
arXiv · 17 iun. 2026
PapersPlease: A Benchmark for Evaluating Motivational Values of Large Language Models Based on ERG Theory
arXiv · 21 iul. 2026
Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models
arXiv · 22 sept. 2026