Are LLM Agents Behaviorally Coherent? A Latent Profiling Approach for Social Simulation

Title:Are LLM Agents Behaviorally Coherent? A Latent Profiling Approach for Social Simulation

Abstract:LLM agents are increasingly proposed as substitutes for human participants in social research, usually on the strength of survey responses that match those of the humans they represent. We argue that such output-level alignment is not enough. Drawing on the behavioral sciences, we propose Latent Profiling, a framework for testing behavioral coherence: whether an agent's conversational behavior follows, as human behavioral models predict, from its own latent profile elicited by short surveys. As a case study, we test six hypotheses about preference, openness and agreement across nine open models. Agents reproduce surface-level trends but systematically fail in-depth tests: they are not behaviorally coherent, even when their individual responses look human. Latent Profiling thus offers a theory-grounded instrument for a behavioral science of LLMs.

19 pages. Project page: this https URL

Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)

arXiv:2509.03736 [cs.AI]

(or arXiv:2509.03736v3 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2509.03736 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.