Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions

arXiv:2605.00226v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly tasked with strategic decision-making under incomplete information, such as in negotiation and policymaking. While LLMs can excel at many such tasks, they also fail in ways that are poorly understood. We shed light on these failures by uncovering two fundamental gaps in the internal mechanisms underlying the decision-making of LLMs in incomplete-information games, supported by experiments with open-weight models Llama 3.1, Qwen3, and gpt-oss. First, an observation-belief gap: LLMs' internal representations of latent game states are substantially more accurate than their own verbal reports. However, these representations, which we call internal beliefs following game-theoretic terminology, are brittle. In particular, the belief accuracy degrades with multi-hop reasoning, exhibits primacy and recency biases, and drifts away from Bayesian coherence over extended interactions. Second, a belief-action gap: The implicit conversion of internal beliefs into actions is weaker than that of the beliefs externalized in the prompt, yet neither belief-conditioning consistently achieves higher game payoffs. Moreover, acting optimally on the decoded beliefs would improve payoffs in about 95% of games, pointing to a bottleneck in the belief-to-action conversion. These results show how analyzing LLMs' internal processes can expose systematic vulnerabilities that warrant caution before deploying LLMs in strategic domains without robust guardrails.

Title:Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions

Abstract:Large language models (LLMs) are increasingly tasked with strategic decision-making under incomplete information, such as in negotiation and policymaking. While LLMs can excel at many such tasks, they also fail in ways that are poorly understood. We shed light on these failures by uncovering two fundamental gaps in the internal mechanisms underlying the decision-making of LLMs in incomplete-information games, supported by experiments with open-weight models Llama 3.1, Qwen3, and gpt-oss. First, an observation-belief gap: LLMs' internal representations of latent game states are substantially more accurate than their own verbal reports. However, these representations, which we call internal beliefs following game-theoretic terminology, are brittle. In particular, the belief accuracy degrades with multi-hop reasoning, exhibits primacy and recency biases, and drifts away from Bayesian coherence over extended interactions. Second, a belief-action gap: The implicit conversion of internal beliefs into actions is weaker than that of the beliefs externalized in the prompt, yet neither belief-conditioning consistently achieves higher game payoffs. Moreover, acting optimally on the decoded beliefs would improve payoffs in about 95% of games, pointing to a bottleneck in the belief-to-action conversion. These results show how analyzing LLMs' internal processes can expose systematic vulnerabilities that warrant caution before deploying LLMs in strategic domains without robust guardrails.

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Science and Game Theory (cs.GT)

arXiv:2605.00226 [cs.CL]

(or arXiv:2605.00226v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2605.00226 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions

Related coverage