analiza aiqbench
Current RLHF alignment methods face fundamental limitations because they aggregate conflicting preferences and rely on binary comparisons that fail to capture nuanced, context-dependent user demands. The study reveals that even widely requested values like truthfulness are defined by incompatible epistemological bases across different user groups, making universal reward models insufficient for addressing the plurality of human needs. These findings suggest that future alignment strategies must move beyond flattening diverse signals into single metrics to effectively resolve issues like hallucination and controversial feature adoption.
What Do People Actually Want From AI? Mapping Preference Plurality
arXiv:2606.06674v1 Announce Type: new Abstract: Large Language Models (LLMs) are often fine-tuned through Reinforcement Learning from Human Feedback (RLHF) to align with people's preferences and values. However, this method has known limitations: it aggregates conflicting preferences, often relies on unrepresentative samples, and uses only binary comparisons. Analysing 1,500 open-ended responses from the PRISM dataset across 75 countries, we examine what people actually want from AI systems and reveal concrete failures of current methods. We find that different people want different things: most values are requested by fewer than a quarter of respondents, with truthfulness the sole exception at 49%. Furthermore, the same words hide divergent meanings: when people describe what they mean by "truthfulness", they reveal distinct, potentially incompatible, epistemological bases, as some ask for sourced claims, some for expert opinions, and some even ask for unpopular views. Certain capabilities, namely how human-like a model behaves, and some features, like AI guardrails, are outright controversial, with some desiring them and others rejecting them. We additionally find that people often use contextual distinctions (what AI should do "by default" versus "if requested") that binary comparisons cannot capture. These findings expose fundamental problems in current alignment practices. When 49% request truthfulness but define it differently, this is unlikely to be captured by a single reward model. The persistence of high hallucination rates in well-funded models, despite users' clear demands for accuracy, suggests that current methods fail to identify actual preferences. This paper sheds light on the situated, contested, imperfect signals that are currently being flattened into universal preference models, a practice others have characterised as epistemic violence.
Title:What Do People Actually Want From AI? Mapping Preference Plurality
Abstract:Large Language Models (LLMs) are often fine-tuned through Reinforcement Learning from Human Feedback (RLHF) to align with people's preferences and values. However, this method has known limitations: it aggregates conflicting preferences, often relies on unrepresentative samples, and uses only binary comparisons. Analysing 1,500 open-ended responses from the PRISM dataset across 75 countries, we examine what people actually want from AI systems and reveal concrete failures of current methods. We find that different people want different things: most values are requested by fewer than a quarter of respondents, with truthfulness the sole exception at 49%. Furthermore, the same words hide divergent meanings: when people describe what they mean by "truthfulness", they reveal distinct, potentially incompatible, epistemological bases, as some ask for sourced claims, some for expert opinions, and some even ask for unpopular views. Certain capabilities, namely how human-like a model behaves, and some features, like AI guardrails, are outright controversial, with some desiring them and others rejecting them. We additionally find that people often use contextual distinctions (what AI should do "by default" versus "if requested") that binary comparisons cannot capture. These findings expose fundamental problems in current alignment practices. When 49% request truthfulness but define it differently, this is unlikely to be captured by a single reward model. The persistence of high hallucination rates in well-funded models, despite users' clear demands for accuracy, suggests that current methods fail to identify actual preferences. This paper sheds light on the situated, contested, imperfect signals that are currently being flattened into universal preference models, a practice others have characterised as epistemic violence.
Computation and Language (cs.CL); Computers and Society (cs.CY)
arXiv:2606.06674 [cs.CL]
(or arXiv:2606.06674v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.06674 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.1145/3805689.3812398 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 379 cuvinte
Subiecte conexe
Pluralistic Alignment in Language Models: Objectives, Human References, and Preference Representations
arXiv · 21 oct. 2026
A Circuit for Plural Reference: How LLMs Represent and Retrieve Singular and Plural Entities
arXiv · 4 sept. 2026
Multiple latent orderings better predict language model preferences
arXiv · 22 sept. 2026
Hidden Consensus:Preference-Validity Compression in Human Feedback
arXiv · 10 iun. 2026