aiqbench analysis
This study identifies the "alignment target problem," revealing that human moral judgments diverge significantly depending on whether an AI's actions are attributed to its own agency or explicitly linked to human designers. The findings suggest that making human design intent visible triggers stricter, rule-based reasoning for both the programmed robot and its creators, challenging current assumptions about using general human behavior as a universal alignment benchmark. For researchers in safety and policy, this highlights the complexity of reconciling pluralistic moral evaluations when developing high-stakes artificial agents.
The Alignment Target Problem: Divergent Moral Judgments of Humans, AI Systems, and Their Designers
arXiv:2604.24155v4 Announce Type: replace-cross Abstract: The project of aligning machine behavior with human values raises a basic problem: whose moral expectations should guide AI decision-making? Much alignment research assumes that the appropriate benchmark is how humans themselves would act in a given situation. Studies of agent-type value forks challenge this assumption by showing that people do not always judge humans and AI systems identically. This paper extends that challenge by examining two further possibilities: first, that evaluations of AI behavior change when its human origins are made visible; and second, that people judge the humans who program AI systems differently from either the machines or the human actors they are compared against. An experiment with 1,002 U.S. adults measured moral judgments in a runaway mine train scenario, varying the subject of evaluation across four conditions: a repairman, a repair robot, a repair robot programmed by company engineers, and company engineers programming a repair robot. We find no significant difference in evaluations of the repairman and the robot. However, judgments shifted substantially when the robot's actions were described as the product of human design. Participants exhibited markedly more deontological, rule-based reasoning when evaluating either the programmed robot or the engineers who programmed it, suggesting that rendering human agency visible activates heightened moral constraints. These findings indicate that people may evaluate humans, AI systems acting in the same situation, and the humans who design them in meaningfully different ways. The fact that these evaluations do not necessarily converge gives rise to the alignment target problem: which normative target should guide the development of artificial moral agents in high-stakes domains, and whether these plural judgments can be reconciled within a coherent account of value alignment.
Abstract:The project of aligning machine behavior with human values raises a basic problem: whose moral expectations should guide AI decision-making? Much alignment research assumes that the appropriate benchmark is how humans themselves would act in a given situation. Studies of agent-type value forks challenge this assumption by showing that people do not always judge humans and AI systems identically. This paper extends that challenge by examining two further possibilities: first, that evaluations of AI behavior change when its human origins are made visible; and second, that people judge the humans who program AI systems differently from either the machines or the human actors they are compared against. An experiment with 1,002 U.S. adults measured moral judgments in a runaway mine train scenario, varying the subject of evaluation across four conditions: a repairman, a repair robot, a repair robot programmed by company engineers, and company engineers programming a repair robot. We find no significant difference in evaluations of the repairman and the robot. However, judgments shifted substantially when the robot's actions were described as the product of human design. Participants exhibited markedly more deontological, rule-based reasoning when evaluating either the programmed robot or the engineers who programmed it, suggesting that rendering human agency visible activates heightened moral constraints. These findings indicate that people may evaluate humans, AI systems acting in the same situation, and the humans who design them in meaningfully different ways. The fact that these evaluations do not necessarily converge gives rise to the alignment target problem: which normative target should guide the development of artificial moral agents in high-stakes domains, and whether these plural judgments can be reconciled within a coherent account of value alignment.
Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)
arXiv:2604.24155 [cs.CY]
(or arXiv:2604.24155v4 [cs.CY] for this version)
From: Benjamin Chen Mr. [view email] [v1] Mon, 27 Apr 2026 08:12:34 UTC (1,909 KB) [v2] Tue, 12 May 2026 10:52:14 UTC (1,909 KB) [v3] Fri, 26 Jun 2026 08:37:14 UTC (2,434 KB) [v4] Wed, 29 Jul 2026 10:57:33 UTC (2,434 KB)
Full text under the publisher's open licence · 337 words
Related coverage
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026
Deconfusing Corrigibility
AI Alignment Forum · Oct 17, 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · Oct 2, 2026
The Alignment Problem: Computer Science Meets Analytic Philosophy
AI Alignment Forum · Oct 18, 2026