Emulate or Estimate? The Divergent Strengths of Base and Post-Trained Language Models for Opinion Simulation

Title:Emulate or Estimate? The Divergent Strengths of Base and Post-Trained Language Models for Opinion Simulation

Abstract:Large language models are increasingly used to simulate human opinions, but prior work reports conflicting results: some studies find promising alignment with human survey data, while others find persona collapse and weak demographic sensitivity. We show that much of this conflict stems from conflating two distinct tasks. We call the first task emulation, in which models generate individual responses that aggregate into a population distribution. We call the second task estimation, in which models directly predict the population distribution. Evaluating six matched base and post-trained models on the Pew American Trends Panel, we find that base models are stronger emulators: they produce response distributions closer to human ground truth and better preserve demographic structure. Post-trained models are stronger estimators, producing more accurate distributional predictions when asked directly. We propose that model selection for human simulation should be guided by whether the task requires generating text or predicting distributions.

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

arXiv:2608.03044 [cs.CL]

(or arXiv:2608.03044v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2608.03044 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.