aiqbench analysis
This study reveals that LLM responses to psychological questionnaires are dominated by a shared latent structure reflecting experiential self-attribution and social desirability rather than genuine human traits. By analyzing 50 models across 45 instruments, the research demonstrates that current psychometric profiles capture response tendencies intrinsic to model architecture and training data, necessitating new interpretive frameworks for AI characterization.
The Shared Latent Structure of LLM Psychometric Responses
arXiv:2605.05080v2 Announce Type: replace Abstract: Human psychometric questionnaires are increasingly used to characterize large language models (LLMs), but it remains unclear what exactly their responses reflect. To answer this question we analyze responses from 50 LLMs across 45 questionnaires spanning multiple psychological domains using psychometric methodology. We found a dominant principal component explaining 47.1% of variance across standardized, response-derived questionnaire-level primary scores, revealing substantial shared structure across instruments. We investigate its meaning through semantic analysis, respondent framing, and human annotations. Semantic analysis suggests that the component partly reflects experiential self-attribution: how readily models endorse statements describing subjective states and bodily experiences. A model score emphasizing items whose disagreement decreases under human simulation correlates strongly with the component ($r=0.864$). Human ratings of 289 matched items show that inner-life content is positively associated, and social desirability negatively associated, with item-component correlations after mutual adjustment and controls for other semantic properties. Together, six rated properties explain 22.1% of variance in Fisher-transformed item-component correlations, supporting a partial interpretation involving both experiential self-attribution and socially desirable responding. These findings suggest that LLM psychometric profiles share response tendencies that require interpretation beyond the human traits their questionnaires were designed to measure.
Abstract:Human psychometric questionnaires are increasingly used to characterize large language models (LLMs), but it remains unclear what exactly their responses reflect. To answer this question we analyze responses from 50 LLMs across 45 questionnaires spanning multiple psychological domains using psychometric methodology. We found a dominant principal component explaining 47.1% of variance across standardized, response-derived questionnaire-level primary scores, revealing substantial shared structure across instruments. We investigate its meaning through semantic analysis, respondent framing, and human annotations. Semantic analysis suggests that the component partly reflects experiential self-attribution: how readily models endorse statements describing subjective states and bodily experiences. A model score emphasizing items whose disagreement decreases under human simulation correlates strongly with the component ($r=0.864$). Human ratings of 289 matched items show that inner-life content is positively associated, and social desirability negatively associated, with item-component correlations after mutual adjustment and controls for other semantic properties. Together, six rated properties explain 22.1% of variance in Fisher-transformed item-component correlations, supporting a partial interpretation involving both experiential self-attribution and socially desirable responding. These findings suggest that LLM psychometric profiles share response tendencies that require interpretation beyond the human traits their questionnaires were designed to measure.
Computation and Language (cs.CL)
arXiv:2605.05080 [cs.CL]
(or arXiv:2605.05080v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.05080 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 283 words
Related coverage
Different representation learning objectives recover distinct latent structures from the same psychometric data
arXiv · Sep 2, 2026
Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacy
arXiv · Aug 27, 2026
Beyond ID Embeddings: Process-Grounded Language Modeling for Cognitive Diagnosis
arXiv · Sep 14, 2026
LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model
arXiv · Aug 25, 2026