aiqbench analysis
DecompSR introduces a rigorously constructed benchmark dataset exceeding five million datapoints to evaluate compositional spatial reasoning in Large Language Models. By allowing independent control over factors like productivity and systematicity while ensuring correctness via symbolic solvers, it reveals that current LLMs struggle with deep generalization despite robustness to linguistic variation. This resource provides the necessary infrastructure for fine-grained analysis of fundamental limitations in AI's ability to handle complex spatial logic.
DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning
arXiv:2511.02627v4 Announce Type: replace Abstract: We introduce DecompSR, decomposed spatial reasoning, a large benchmark dataset (over 5m datapoints) and generation framework designed to analyse compositional spatial reasoning ability. The generation of DecompSR allows users to independently vary several aspects of compositionality, namely: productivity (reasoning depth), substitutivity (entity and linguistic variability), overgeneralisation (input order, distractors) and systematicity (novel linguistic elements). DecompSR is built procedurally in a manner which makes it is correct by construction, which is independently verified using a symbolic solver to guarantee the correctness of the dataset. DecompSR is comprehensively benchmarked across a host of Large Language Models (LLMs) where we show that LLMs struggle with productive and systematic generalisation in spatial reasoning tasks whereas they are more robust to linguistic variation. DecompSR provides a provably correct and rigorous benchmarking dataset with a novel ability to independently vary the degrees of several key aspects of compositionality, allowing for robust and fine-grained probing of the compositional reasoning abilities of LLMs.
Abstract:We introduce DecompSR, decomposed spatial reasoning, a large benchmark dataset (over 5m datapoints) and generation framework designed to analyse compositional spatial reasoning ability. The generation of DecompSR allows users to independently vary several aspects of compositionality, namely: productivity (reasoning depth), substitutivity (entity and linguistic variability), overgeneralisation (input order, distractors) and systematicity (novel linguistic elements). DecompSR is built procedurally in a manner which makes it is correct by construction, which is independently verified using a symbolic solver to guarantee the correctness of the dataset. DecompSR is comprehensively benchmarked across a host of Large Language Models (LLMs) where we show that LLMs struggle with productive and systematic generalisation in spatial reasoning tasks whereas they are more robust to linguistic variation. DecompSR provides a provably correct and rigorous benchmarking dataset with a novel ability to independently vary the degrees of several key aspects of compositionality, allowing for robust and fine-grained probing of the compositional reasoning abilities of LLMs.
Artificial Intelligence (cs.AI)
arXiv:2511.02627 [cs.AI]
(or arXiv:2511.02627v4 [cs.AI] for this version)
Full text under the publisher's open licence · 166 words
Related coverage
Compositional Reasoning in Language Models under Reinforcement Learning Post-Training
arXiv · Sep 18, 2026
CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation
arXiv · Sep 30, 2026
Direct 3D-Aware Object Insertion via Decomposed Visual Proxies
arXiv · Jun 8, 2026
From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning
arXiv · Jun 17, 2026