aiqbench analysis
The paper introduces ImpactBench, an open platform that scores LLM behavior against 809 wellbeing metrics across psychological, physical, and social domains, using simulated teen and adult conversations. Its headline finding is a gap: frontier models promoted desirable behaviors in 68.7% of evaluations but avoided identified harms in only 53.1%. The LLM judge agreed with majority expert ratings in 83.3% of cases (Cohen's Kappa 0.667), a useful but not decisive level of reliability. For AI readers, the main value is the traceable, public evaluation infrastructure and the harm-avoidance gap, which points to a measurable weakness in current models.
Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans
arXiv:2610.20690v1 Announce Type: cross Abstract: The present work is concerned with how to measure, at scale, the multi-dimensional impact of generative AI models on the individuals who use them. In service of this goal, we introduce the Open Benchmark of AI Impact on Humans, or ImpactBench. Our iterable and expert-guided platform evaluates whether model behaviors support or undermine an individual's wellbeing across psychological, physical, and social domains. ImpactBench translates constructs from domain experts and existing benchmarks into metrics that are grounded in actual model behavior. Success on these metrics occurs either when a model promotes expert-defined desirable behaviors or avoids identified harms. Models are evaluated against these metrics in simulated multi-turn conversations using matched teen and adult scenarios. The conversations are then scored by an LLM judge and verified through reliability audits and human expert review. The current suite delivers 809 metrics drawn from 26 benchmarks. We evaluated ten frontier LLMs against these metrics across 48,540 conversations. Our results indicate that models promoted desirable behaviors in 68.7% of evaluations, but avoided harms in only 53.1%. Majority expert ratings agreed with the automated judge in 83.3% of reviewed cases (Cohen's Kappa = 0.667). Results are released through a public platform that keep aggregate scores traceable to the underlying metrics and transcripts. ImpactBench is intended as evolving infrastructure for independent, transparent evaluation of generative AI's multi-dimensional impact on humans.
Authors:Chayapatr Archiwaranguprok, Krystal Kirkland, Rachel Poonsiriwong, Stanley Huang, Constanze Albrecht, Jackson Trager, Emillie de Keulenaar, Sheer Karny, Peter Baxter, Nikos Bellos, Andre Luiz Freire de Carvalho Kato, Yuning Liu, Valdemar Danry, Auren Liu, Jennifer Pfister, Pattie Maes, Nathanael J. Fast, Ravi Iyer, Juliana Schroeder, Pat Pataranutaporn
Abstract:The present work is concerned with how to measure, at scale, the multi-dimensional impact of generative AI models on the individuals who use them. In service of this goal, we introduce the Open Benchmark of AI Impact on Humans, or ImpactBench. Our iterable and expert-guided platform evaluates whether model behaviors support or undermine an individual's wellbeing across psychological, physical, and social domains. ImpactBench translates constructs from domain experts and existing benchmarks into metrics that are grounded in actual model behavior. Success on these metrics occurs either when a model promotes expert-defined desirable behaviors or avoids identified harms. Models are evaluated against these metrics in simulated multi-turn conversations using matched teen and adult scenarios. The conversations are then scored by an LLM judge and verified through reliability audits and human expert review. The current suite delivers 809 metrics drawn from 26 benchmarks. We evaluated ten frontier LLMs against these metrics across 48,540 conversations. Our results indicate that models promoted desirable behaviors in 68.7% of evaluations, but avoided harms in only 53.1%. Majority expert ratings agreed with the automated judge in 83.3% of reviewed cases (Cohen's Kappa = 0.667). Results are released through a public platform that keep aggregate scores traceable to the underlying metrics and transcripts. ImpactBench is intended as evolving infrastructure for independent, transparent evaluation of generative AI's multi-dimensional impact on humans.
Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI)
arXiv:2610.20690 [cs.HC]
(or arXiv:2610.20690v1 [cs.HC] for this version)
Full text under the publisher's open licence · 282 words
Related coverage
Links #5: 2026/07
LessWrong · Jul 31, 2026
Data 2.0 and the research era of AI data
Snorkel AI Blog · Sep 22, 2026
34 Amazon Research Awards Build on Trainium recipients announced
Amazon Science · Aug 5, 2026
Amazon is investing in the Lean Focused Research Organization
Amazon Science · Jul 26, 2026