aiqbench analysis
This paper is squarely about AI: it evaluates the strategic reasoning of large language models using procedurally generated imperfect-information betting games, which addresses the contamination and saturation problems of fixed benchmarks. The useful signal for readers is the finding that the performance gap between models widens on the hardest games, and that factors like thinking budget and matching reasoning to a game's demands correlate with better outcomes. Since the results are correlational ablations from a single tournament, they point toward a promising evaluation method rather than settling which models reason strategically best.
GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models
arXiv:2605.23238v2 Announce Type: replace-cross Abstract: Fixed game suites for evaluating strategic reasoning in large language models (LLMs) saturate and are hard to keep free of contamination. We introduce GENSTRAT, a system for evaluating the strategic capabilities of LLMs driven by procedural generation of two-player zero-sum imperfect-information betting games, allowing an evaluator to draw novel, fresh games on demand. Each game is characterized on a scale of five complexity axes (state space, temporal contingency, opponent dependence, private information, precision) and measured by Monte Carlo simulation. Using this generator, we construct a 50-game benchmark and hold a tournament of twelve frontier and open-weight models, gathering data from over 40,000 LLM vs. LLM matches. We further construct a composite complexity axis, and find that, on the eight hardest games, the gap between the best and worst model is 1.7x its value across the whole set of games. Ablations show that a higher thinking budget, disclosing the opponent's strategy, and higher reasoning matched to a game's specific demands (for private information and planning depth) were associated with higher margins for the LLM player.
Abstract:Fixed game suites for evaluating strategic reasoning in large language models (LLMs) saturate and are hard to keep free of contamination. We introduce GENSTRAT, a system for evaluating the strategic capabilities of LLMs driven by procedural generation of two-player zero-sum imperfect-information betting games, allowing an evaluator to draw novel, fresh games on demand. Each game is characterized on a scale of five complexity axes (state space, temporal contingency, opponent dependence, private information, precision) and measured by Monte Carlo simulation. Using this generator, we construct a 50-game benchmark and hold a tournament of twelve frontier and open-weight models, gathering data from over 40,000 LLM vs. LLM matches. We further construct a composite complexity axis, and find that, on the eight hardest games, the gap between the best and worst model is 1.7x its value across the whole set of games. Ablations show that a higher thinking budget, disclosing the opponent's strategy, and higher reasoning matched to a game's specific demands (for private information and planning depth) were associated with higher margins for the LLM player.
Artificial Intelligence (cs.AI); Computer Science and Game Theory (cs.GT); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
arXiv:2605.23238 [cs.AI]
(or arXiv:2605.23238v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2605.23238 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 278 words
Related coverage
SAGE: Structured Strategic Reasoning for Efficient LLM Game Playing
arXiv · Sep 29, 2026
STRIDE: Strategic Trajectory Reasoning via Discriminative Estimation for Verifiable Reinforcement Learning
arXiv · Jun 16, 2026
What Counts as Strategic Reasoning? A Systematic Mapping of Chess Research on Humans, Engines, and Language Models
arXiv · Sep 17, 2026
Poker Arena: Multi-Axis Profiling of Strategic Reasoning and Memory in LLMs
arXiv · Jun 15, 2026