aiqbench analysis
This research investigates whether reasoning-focused LLMs possess genuine Theory of Mind capabilities or simply exhibit improved robustness against prompt variations and task perturbations. The findings suggest that gains in these benchmarks stem from a general ability to reach correct answers under diverse conditions rather than the emergence of a specific psychological faculty, challenging recent claims about advanced social cognition in models.
Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
arXiv:2608.04646v1 Announce Type: new Abstract: Large language models (LLMs) have recently shown strong performance on Theory of Mind (ToM) tests, prompting debate about the nature and validity of the underlying capabilities. At the same time, reasoning-oriented LLMs trained via reinforcement learning with verifiable rewards have demonstrated notable improvements across a range of benchmarks. In this work, we examine the behavior of such reasoning models in ToM tasks using novel adaptations of machine psychological experiments together with results from established benchmarks. We observe that reasoning models consistently exhibit increased robustness to prompt variations and task perturbations. Our analysis suggests these gains come at least partly from models being more robust at reaching the correct answer under prompt and task variation. We read this as evidence for a robustness-based account rather than for a new ToM-specific ability.
Abstract:Large language models (LLMs) have recently shown strong performance on Theory of Mind (ToM) tests, prompting debate about the nature and validity of the underlying capabilities. At the same time, reasoning-oriented LLMs trained via reinforcement learning with verifiable rewards have demonstrated notable improvements across a range of benchmarks. In this work, we examine the behavior of such reasoning models in ToM tasks using novel adaptations of machine psychological experiments together with results from established benchmarks. We observe that reasoning models consistently exhibit increased robustness to prompt variations and task perturbations. Our analysis suggests these gains come at least partly from models being more robust at reaching the correct answer under prompt and task variation. We read this as evidence for a robustness-based account rather than for a new ToM-specific ability.
Computation and Language (cs.CL)
arXiv:2608.04646 [cs.CL]
(or arXiv:2608.04646v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.04646 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 225 words
Related coverage
How Would We Know? Reflections on trying to make AI go well amid deep uncertainty
LessWrong · Oct 5, 2026
The basic argument for making AI better at moral philosophy
LessWrong · Oct 14, 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026
Links #3: 2026/06 Part 1
LessWrong · Jun 15, 2026