aiqbench analysis
This study demonstrates that automated cognitive theories derived from imperfect behavioral simulators can successfully generalize to human data, validating a new approach for scaling scientific discovery without relying solely on costly human trials. By using LLM agents to design and arbitrate experiments within a closed loop, the system identified robust decision-making theories that outperformed canonical models on held-out tests. This work bridges the gap between simulated environments and real-world human behavior, offering a scalable pathway for advancing cognitive science through AI-driven experimentation.
Sparks of In Silico Cognitive Science: Theories from Simulated Data Can Generalize to Humans
arXiv:2609.08003v1 Announce Type: new Abstract: Behavioral foundation models have been proposed as stand-ins for human participants across settings, but it is unclear whether theories discovered on them generalize to humans or merely characterize the simulator. We ran the Automated Cognitive Scientist (\textsc{AutoCog}), a closed-loop discovery system in which LLM agents design theory-discriminating experiments, collect responses, arbitrate between competing theories, and synthesize successors, entirely on behavior simulated by Centaur, a foundation model of human behavior. In a multi-attribute decision-making setting, the theories \textsc{AutoCog} found on Centaur generalized to human data: they outperformed canonical theories on ten held-out experiments and were rivaled only by theories found by running the same loop on people. We argue that this succeeds despite the simulator's inevitable imperfections because a discovery loop that arbitrates between competing theories demands less of its simulator than estimation does. The simulator only needs to capture the regularities that distinguish the theories, and not necessarily reproduce behavior precisely. Imperfect simulators can therefore widen the search over theories, with human data then testing whether the surfaced theories generalize.
Abstract:Behavioral foundation models have been proposed as stand-ins for human participants across settings, but it is unclear whether theories discovered on them generalize to humans or merely characterize the simulator. We ran the Automated Cognitive Scientist (\textsc{AutoCog}), a closed-loop discovery system in which LLM agents design theory-discriminating experiments, collect responses, arbitrate between competing theories, and synthesize successors, entirely on behavior simulated by Centaur, a foundation model of human behavior. In a multi-attribute decision-making setting, the theories \textsc{AutoCog} found on Centaur generalized to human data: they outperformed canonical theories on ten held-out experiments and were rivaled only by theories found by running the same loop on people. We argue that this succeeds despite the simulator's inevitable imperfections because a discovery loop that arbitrates between competing theories demands less of its simulator than estimation does. The simulator only needs to capture the regularities that distinguish the theories, and not necessarily reproduce behavior precisely. Imperfect simulators can therefore widen the search over theories, with human data then testing whether the surfaced theories generalize.
Artificial Intelligence (cs.AI)
arXiv:2609.08003 [cs.AI]
(or arXiv:2609.08003v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.08003 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 264 words
Related coverage
A Multi-Stage Rule-Chaining Framework for Compositional and Interpretable Cognitive Reasoning
arXiv · Sep 12, 2026
Cognitive Trajectory Modeling: Quantifying Human-AI Co-Creation through Cognitively Grounded Interaction Trajectories
arXiv · Jun 16, 2026
A Taxonomy of Cognitive Capability Gaps in Generative and Agentic AI
arXiv · Aug 5, 2026
EGO: a Recursive and Self-Referential Cognitive Architecture for Artificial General Intelligence
arXiv · Jul 20, 2026