aiqbench analysis
This survey treats preference-based alignment as a problem of collective decision-making: when annotators disagree, the reward model effectively picks a winner, and social choice theory offers tools to see how that choice is made. For AI safety and alignment readers, the practical takeaway is that aggregation is a design decision with its own failure modes, not a neutral preprocessing step. The paper's value lies in mapping that design space, though its claims are conceptual and survey-based rather than new empirical results.
AI Alignment From Social Choice Perspectives
arXiv:2606.21550v2 Announce Type: replace Abstract: Alignment from human feedback uses human judgments about model outputs to steer the behavior of language models after pretraining. When those judgments reflect conflicting views of desirable behavior, the learned objective becomes an aggregate determination of what the model should prefer. We survey recent work that has studied this aggregation problem through the lens of social choice theory. We illustrate how the social choice perspective helps identify failure modes in the feedback aggregation layer and reveals a broader design space for handling disagreement in explicit and principled ways.
Abstract:Alignment from human feedback uses human judgments about model outputs to steer the behavior of language models after pretraining. When those judgments reflect conflicting views of desirable behavior, the learned objective becomes an aggregate determination of what the model should prefer. We survey recent work that has studied this aggregation problem through the lens of social choice theory. We illustrate how the social choice perspective helps identify failure modes in the feedback aggregation layer and reveals a broader design space for handling disagreement in explicit and principled ways.
Artificial Intelligence (cs.AI)
arXiv:2606.21550 [cs.AI]
(or arXiv:2606.21550v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.21550 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 180 words
Related coverage
Hint-based CoT faithfulness evals still mostly work on Claude
LessWrong · Jul 30, 2026
The Alignment Problem: Computer Science Meets Analytic Philosophy
LessWrong · Oct 18, 2026
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · Sep 18, 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026