aiqbench analysis
This article is an arXiv paper on LLM generalization versus memorization, using chess positions as a controlled testbed. Its central finding is that LLM performance degrades as positions become less covered by training priors, with models often proposing illegal moves in unfamiliar states. For AI readers, the value lies in the methodology: a way to probe reasoning without access to training data, and evidence that scale alone may not deliver systematic generalization.
Disentangling generalization and memorization in large language models using chess
arXiv:2601.16823v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) exhibit remarkable capabilities, yet it remains unclear to what extent these reflect sophisticated recall or genuine reasoning ability. We introduce chess as a controlled testbed aimed at disentangling these faculties. Leveraging the game's structure and scalable engine evaluations, we construct a taxonomy of positions varying in density of relevant priors -- ranging from common states solvable by memorization to completely novel ones requiring generalization. Crucially, our approach achieves this distinction without requiring explicit knowledge of the models' training data. Applying this taxonomy, we combine a longitudinal analysis of the GPT lineage with a rigorous evaluation of contemporary models, including Claude Opus and Gemini. Our analysis reveals a steep gradient: performance consistently degrades as the density of relevant priors decreases. Notably, for tasks with few relevant priors, base model performance regresses to the random-play baseline, and all models propose illegal moves at sharply increasing rates, indicating that they fail to reliably apply the fixed rules of the game in unfamiliar positions. While newer models improve, progress slows significantly for tasks with sparse priors. Furthermore, while reasoning-augmented inference improves performance, its relative marginal benefit per token decreases in the absence of relevant priors. These results suggest limitations in systematic generalization, highlighting the need for mechanisms beyond scale to achieve robust performance when deprived of relevant priors.
Abstract:Large Language Models (LLMs) exhibit remarkable capabilities, yet it remains unclear to what extent these reflect sophisticated recall or genuine reasoning ability. We introduce chess as a controlled testbed aimed at disentangling these faculties. Leveraging the game's structure and scalable engine evaluations, we construct a taxonomy of positions varying in density of relevant priors -- ranging from common states solvable by memorization to completely novel ones requiring generalization. Crucially, our approach achieves this distinction without requiring explicit knowledge of the models' training data. Applying this taxonomy, we combine a longitudinal analysis of the GPT lineage with a rigorous evaluation of contemporary models, including Claude Opus and Gemini. Our analysis reveals a steep gradient: performance consistently degrades as the density of relevant priors decreases. Notably, for tasks with few relevant priors, base model performance regresses to the random-play baseline, and all models propose illegal moves at sharply increasing rates, indicating that they fail to reliably apply the fixed rules of the game in unfamiliar positions. While newer models improve, progress slows significantly for tasks with sparse priors. Furthermore, while reasoning-augmented inference improves performance, its relative marginal benefit per token decreases in the absence of relevant priors. These results suggest limitations in systematic generalization, highlighting the need for mechanisms beyond scale to achieve robust performance when deprived of relevant priors.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2601.16823 [cs.CL]
(or arXiv:2601.16823v3 [cs.CL] for this version)
From: Leonard S. Pleiss [view email] [v1] Fri, 23 Jan 2026 15:23:08 UTC (537 KB) [v2] Tue, 19 May 2026 12:03:44 UTC (698 KB) [v3] Tue, 20 Oct 2026 21:13:38 UTC (700 KB)
Full text under the publisher's open licence · 266 words
Related coverage
Self-Supervised Consistency Enhanced Disentangled Learning for Neural Decoding Generalization in Brain-Machine Interface
arXiv · Jul 28, 2026
Circuit mechanisms for disentangled representations and generalization in the hippocampus
Nature · Oct 27, 2026
Fix the Mind, Not the Move: Interpretable AI Assistance via Knowledge-Gap Localization
arXiv · Jun 6, 2026
Coherence-Guided Pyramid Self-Contrastive Disentanglement for Ultrasound Imaging via Single-Shot Test-Time Training
arXiv · Oct 27, 2026