aiqbench analysis
This paper challenges standard LLM alignment strategies by arguing that aggregating human preferences into a single reward signal fails to capture individual values and contextual nuances. The authors propose a shift toward personalized preference learning, noting that while this introduces risks like filter bubbles, bounded frameworks can balance individual autonomy with collective safety. The work outlines a necessary research agenda to move beyond optimizing for a hypothetical "average user" toward models that respect genuine demographic diversity.
Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences
arXiv:2606.07629v1 Announce Type: new Abstract: Current approaches to aligning large language models (LLMs) aggregate diverse human preferences into a single reward signal, effectively optimizing for a hypothetical ``average user'' who represents no real person particularly well. This position paper argues that LLMs should learn personalized, individual preferences rather than aggregated ones. We show that aggregation masks critical information about preference diversity, individual values, and contextual dependencies, which is a limitation both theoretically grounded in social choice theory and empirically evident across demographic groups. We analyze the rich structure that human preferences encode, survey technical approaches to personalization, and systematically address counterarguments on scalability, shared standards, and manipulation risk. While personalization introduces genuine safety challenges including filter bubbles, value lock-in, and psychological manipulation, we argue these are manageable through bounded personalization frameworks that preserve universal safety constraints while accommodating legitimate individual variation. We conclude with a concrete research and policy agenda for developing preference-aware models that respect both individual autonomy and collective safety.
Abstract:Current approaches to aligning large language models (LLMs) aggregate diverse human preferences into a single reward signal, effectively optimizing for a hypothetical ``average user'' who represents no real person particularly well. This position paper argues that LLMs should learn personalized, individual preferences rather than aggregated ones. We show that aggregation masks critical information about preference diversity, individual values, and contextual dependencies, which is a limitation both theoretically grounded in social choice theory and empirically evident across demographic groups. We analyze the rich structure that human preferences encode, survey technical approaches to personalization, and systematically address counterarguments on scalability, shared standards, and manipulation risk. While personalization introduces genuine safety challenges including filter bubbles, value lock-in, and psychological manipulation, we argue these are manageable through bounded personalization frameworks that preserve universal safety constraints while accommodating legitimate individual variation. We conclude with a concrete research and policy agenda for developing preference-aware models that respect both individual autonomy and collective safety.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); Human-Computer Interaction (cs.HC)
arXiv:2606.07629 [cs.LG]
(or arXiv:2606.07629v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.07629 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 189 words
Related coverage
PACIFIC: Can LLMs Discern the Psychometric Traits Influencing Your Preferences? Personality-Driven Preference Alignment in LLMs
arXiv · Sep 14, 2026
Gradient-Aligned Pair Selection for Personalized Preference Optimization
arXiv · Oct 2, 2026
Group Preference Collapse in Personalized Multimodal Large Language Models
arXiv · Jul 28, 2026
From Static Personal Values to Contextualized Personalization: Bayesian Personalized Value Alignment for LLMs
arXiv · Sep 25, 2026