analiza aiqbench
DecompSR introduces a rigorously constructed benchmark of over 5 million datapoints that enables fine-grained probing of compositional spatial reasoning in LLMs. By allowing independent variation of factors like productivity and systematicity, the dataset reveals that current models struggle with reasoning depth and generalization while remaining robust to linguistic changes. This resource offers a critical tool for researchers aiming to stress-test the limits of spatial reasoning capabilities in next-generation AI systems.
DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning
arXiv:2511.02627v3 Announce Type: replace Abstract: We introduce DecompSR, decomposed spatial reasoning, a large benchmark dataset (over 5m datapoints) and generation framework designed to analyse compositional spatial reasoning ability. The generation of DecompSR allows users to independently vary several aspects of compositionality, namely: productivity (reasoning depth), substitutivity (entity and linguistic variability), overgeneralisation (input order, distractors) and systematicity (novel linguistic elements). DecompSR is built procedurally in a manner which makes it is correct by construction, which is independently verified using a symbolic solver to guarantee the correctness of the dataset. DecompSR is comprehensively benchmarked across a host of Large Language Models (LLMs) where we show that LLMs struggle with productive and systematic generalisation in spatial reasoning tasks whereas they are more robust to linguistic variation. DecompSR provides a provably correct and rigorous benchmarking dataset with a novel ability to independently vary the degrees of several key aspects of compositionality, allowing for robust and fine-grained probing of the compositional reasoning abilities of LLMs.
Abstract:We introduce DecompSR, decomposed spatial reasoning, a large benchmark dataset (over 5m datapoints) and generation framework designed to analyse compositional spatial reasoning ability. The generation of DecompSR allows users to independently vary several aspects of compositionality, namely: productivity (reasoning depth), substitutivity (entity and linguistic variability), overgeneralisation (input order, distractors) and systematicity (novel linguistic elements). DecompSR is built procedurally in a manner which makes it is correct by construction, which is independently verified using a symbolic solver to guarantee the correctness of the dataset. DecompSR is comprehensively benchmarked across a host of Large Language Models (LLMs) where we show that LLMs struggle with productive and systematic generalisation in spatial reasoning tasks whereas they are more robust to linguistic variation. DecompSR provides a provably correct and rigorous benchmarking dataset with a novel ability to independently vary the degrees of several key aspects of compositionality, allowing for robust and fine-grained probing of the compositional reasoning abilities of LLMs.
Artificial Intelligence (cs.AI)
arXiv:2511.02627 [cs.AI]
(or arXiv:2511.02627v3 [cs.AI] for this version)
Text integral sub licența deschisă a publicației · 166 cuvinte
Subiecte conexe
Compositional Reasoning in Language Models under Reinforcement Learning Post-Training
arXiv · 18 sept. 2026
CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation
arXiv · 30 sept. 2026
Direct 3D-Aware Object Insertion via Decomposed Visual Proxies
arXiv · 8 iun. 2026
From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning
arXiv · 17 iun. 2026