aiqbench analysis
This study demonstrates that LLM self-report metrics derived from human personality inventories fail to predict actual model behavior, revealing a significant gap between what models claim and how they act. The research identifies verbosity as the only partially reliable self-reported trait, while other factors like responsiveness show weak correlation with human judgments despite some alignment with LLM judges. These findings are critical for developers seeking to build trustworthy systems, as they indicate that standard introspection prompts cannot be relied upon to gauge model safety or alignment without direct behavioral observation.
An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models
arXiv:2606.09843v4 Announce Type: replace-cross Abstract: Do large language models' (LLMs') answers to self-report questionnaires predict how they behave? Prior work finds they do not, but it uses human personality inventories, so the gap could reflect borrowed human constructs rather than LLM self-report itself. We test this with a self-report instrument built from LLM-specific behaviors (e.g., over-refusal, unsolicited disclaimers) whose structure is derived bottom-up. Administering 300 items 30 times to 25 LLMs from 17 developers yields five replicable, reliable factors (Tucker $\phi \geq .957$, $\alpha \geq .930$). We compare these self-reports with 2,500 open-ended behavioral samples rated by 151 humans and an LLM-judge ensemble. Humans and judges agree about model behavior ($\bar{r} = .51$), but self-report barely tracks human ratings ($\bar{r} = .09$, 95% CI $[-.07, .18]$) or rater-free text measures, and correcting for criterion unreliability leaves four of five factors near zero. Verbosity is the partial exception ($r = .40$, 71% of its reliability ceiling). On Responsiveness, self-report tracks LLM judges more than humans ($r = .53$ vs. $.18$; Steiger $p = .04$), and controlling for length and formatting does not remove this: agreement between LLM judges and LLM self-report is weak evidence that either tracks human judgment.
Abstract:Do large language models' (LLMs') answers to self-report questionnaires predict how they behave? Prior work finds they do not, but it uses human personality inventories, so the gap could reflect borrowed human constructs rather than LLM self-report itself. We test this with a self-report instrument built from LLM-specific behaviors (e.g., over-refusal, unsolicited disclaimers) whose structure is derived bottom-up. Administering 300 items 30 times to 25 LLMs from 17 developers yields five replicable, reliable factors (Tucker $\phi \geq .957$, $\alpha \geq .930$). We compare these self-reports with 2,500 open-ended behavioral samples rated by 151 humans and an LLM-judge ensemble. Humans and judges agree about model behavior ($\bar{r} = .51$), but self-report barely tracks human ratings ($\bar{r} = .09$, 95% CI $[-.07, .18]$) or rater-free text measures, and correcting for criterion unreliability leaves four of five factors near zero. Verbosity is the partial exception ($r = .40$, 71% of its reliability ceiling). On Responsiveness, self-report tracks LLM judges more than humans ($r = .53$ vs. $.18$; Steiger $p = .04$), and controlling for length and formatting does not remove this: agreement between LLM judges and LLM self-report is weak evidence that either tracks human judgment.
Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2606.09843 [cs.HC]
(or arXiv:2606.09843v4 [cs.HC] for this version)
From: Juan Manuel Contreras Ph.D. [view email] [v1] Fri, 24 Apr 2026 04:42:09 UTC (577 KB) [v2] Thu, 25 Jun 2026 15:23:14 UTC (571 KB) [v3] Tue, 7 Jul 2026 00:32:51 UTC (579 KB) [v4] Wed, 7 Oct 2026 02:52:07 UTC (598 KB)
Full text under the publisher's open licence · 254 words
Related coverage
An LLM-Native Psychometric Instrument Does Not Predict LLM Behavior: Evidence Across 25 Models
arXiv · Jun 10, 2026
News
Quantum Computing Report · Jun 7, 2026
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026
Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior
arXiv · Jun 12, 2026