aiqbench analysis
This scoping review of 223 studies highlights a critical gap in open-ended wargaming research, where only ~9% grant language models dual control over both player actions and adjudication. The analysis underscores that fidelity in such simulations depends not merely on plausible agent behavior but on the model's ability to reliably function as an impartial world model or arbiter of consequences. Researchers must explicitly define these distinct roles before treating generated transcripts as accurate social simulations, moving beyond surface-level behavioral signals to understand underlying control architectures.
Shall We Play a Game? Language Models for Open-ended Wargames
arXiv:2509.17192v3 Announce Type: replace Abstract: LLM-based social simulations can make a generated transcript look like a single behavioral signal, but the model behind that transcript may be doing several different jobs: choosing what an actor says or does, deciding what happens after an action, or both. The difference matters especially in open-ended wargames, where models are prized for handling unusual actions and ambiguous consequences. We report a scoping review of 223 de-duplicated AI-in-wargames and strategic-simulation papers retrieved through May 1, 2026, describing each simulation by its model-control profile: whether the language model has open-ended control over player actions, adjudication, or both. Only 20 of 223 studies (~9%) give language models both roles. Before treating LM outputs as social simulations, researchers need to know how much creative control the model has over actions and consequences. For open-ended simulations, fidelity depends not only on whether agents behave plausibly, but also on whether language models can reliably act as adjudicators or world models.
Authors:Glenn Matlin, Isaac Song, Yixiong Hao, Parv Mahajan, Evan Montoya, Ryan Bard, Stuart R. Topp, Anthony Wen-Ming Zang, Mohammed Rehan Parwani, Soham Shetty, Mark Riedl
Abstract:LLM-based social simulations can make a generated transcript look like a single behavioral signal, but the model behind that transcript may be doing several different jobs: choosing what an actor says or does, deciding what happens after an action, or both. The difference matters especially in open-ended wargames, where models are prized for handling unusual actions and ambiguous consequences. We report a scoping review of 223 de-duplicated AI-in-wargames and strategic-simulation papers retrieved through May 1, 2026, describing each simulation by its model-control profile: whether the language model has open-ended control over player actions, adjudication, or both. Only 20 of 223 studies (~9%) give language models both roles. Before treating LM outputs as social simulations, researchers need to know how much creative control the model has over actions and consequences. For open-ended simulations, fidelity depends not only on whether agents behave plausibly, but also on whether language models can reliably act as adjudicators or world models.
49 pages (9-page body), 3 figures. Published at the Social Sim'26 Workshop at COLM 2026 (non-archival)
Artificial Intelligence (cs.AI)
arXiv:2509.17192 [cs.AI]
(or arXiv:2509.17192v3 [cs.AI] for this version)
Full text under the publisher's open licence · 207 words
Related coverage
Academia is for Ambition — Alex Zhang, MIT
Latent Space · Oct 2, 2026
Humanity’s Last Invention — Richard Socher of Recursive
Latent Space · Sep 14, 2026
Frontier post-training recipe review with Finbarr Timbers
Interconnects · Jun 16, 2026
Game-Theoretic Disempowerment: Loss of Control to Agencyless AI
LessWrong · Sep 27, 2026