aiqbench analysis
This research establishes a rigorous psychometric framework for evaluating Large Language Models using situational judgment tests to uncover stable behavioral tendencies rather than superficial variations. By leveraging multidimensional item response theory and synthetic personas, the study provides reliable metrics that outperform classical self-report approaches in assessing model consistency across contexts. The released datasets offer valuable resources for advancing objective benchmarks in AI behavior analysis.
Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests
arXiv:2510.22170v3 Announce Type: replace Abstract: Persona conditioning is widely used to steer large language model (LLM) behavior, but it is unclear whether it induces stable behavioral structure or superficial variation. We propose a framework to measure consistent behavioral tendencies using situational judgment tests (SJTs), multidimensional item response theory (MIRT), and structured synthetic personas, treating responses as observations of latent behavioral variables. Across large-scale SJT and persona datasets, we find that persona-conditioned behaviors are stable across runs, latent trait scores predict external benchmarks (e.g., TruthfulQA, EmoBench), and MIRT reveals consistent latent structure. We validate these results through human annotation, benchmark evaluation, and internal consistency analyses. We interpret these traits not as human personality, but as stable behavioral tendencies expressed across contexts. Our results show that scenario-based psychometric evaluation provides a more reliable alternative to classical self-report approaches for assessing LLM behavior, and we release datasets to support further study.
Abstract:Persona conditioning is widely used to steer large language model (LLM) behavior, but it is unclear whether it induces stable behavioral structure or superficial variation. We propose a framework to measure consistent behavioral tendencies using situational judgment tests (SJTs), multidimensional item response theory (MIRT), and structured synthetic personas, treating responses as observations of latent behavioral variables. Across large-scale SJT and persona datasets, we find that persona-conditioned behaviors are stable across runs, latent trait scores predict external benchmarks (e.g., TruthfulQA, EmoBench), and MIRT reveals consistent latent structure. We validate these results through human annotation, benchmark evaluation, and internal consistency analyses. We interpret these traits not as human personality, but as stable behavioral tendencies expressed across contexts. Our results show that scenario-based psychometric evaluation provides a more reliable alternative to classical self-report approaches for assessing LLM behavior, and we release datasets to support further study.
100 pages. Code, and link to datasets here: this https URL
Artificial Intelligence (cs.AI)
I.2.7; I.2.6; H.1.2; J.4
arXiv:2510.22170 [cs.AI]
(or arXiv:2510.22170v3 [cs.AI] for this version)
Full text under the publisher's open licence · 169 words
Related coverage
Deconfusing Corrigibility
AI Alignment Forum · Oct 17, 2026
Measuring Reward-Seeking via Contrastive Belief Updates
LessWrong · Jul 21, 2026
Game-Theoretic Disempowerment: Loss of Control to Agencyless AI
LessWrong · Sep 27, 2026
Evidence about risk should be transparent
LessWrong · Sep 25, 2026