aiqbench analysis
This study challenges recent claims about Large Language Model reasoning failures by demonstrating that statistical artifacts in benchmark datasets, specifically integer distribution shifts, can mimic genuine capability drops. By applying rigorous mixed-effects modeling, the authors reveal that many reported performance issues are not intrinsic model limitations but rather artifacts of specific problem formulations and large-number effects. These findings urge the AI research community to adopt more robust statistical standards before drawing mechanistic conclusions about LLM reasoning capabilities.
The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic
arXiv:2605.28700v3 Announce Type: replace Abstract: The GSM-Symbolic benchmark (Mirzadeh et al., 2025) reported consistent performance drops across 25 Large Language Models (LLMs) when tested on template-generated variants of GSM8K problems, concluding that the models lack genuine reasoning capabilities. We argue that this conclusion rests on shaky statistical ground. Re-evaluating 20 open-weight models using bootstrapped Generalised Linear Mixed Models with per-question random effects, we find that only 8 exhibit statistically significant performance changes under the original prompt format. Moreover, we identify a previously unacknowledged factor: the distribution of integers in problem texts of the main GSM-Symbolic dataset is systematically shifted towards larger values relative to the original GSM8K (K-S statistic = 0.12, p < 0.001), contradicting the original authors' claims. Controlling for this large-number effect accounts for significance in half of the remaining cases. Among models with statistically significant performance deltas, we identify distinct, model-specific behavioural failure profiles -- including fragility of variable binding, arithmetic limitations, and dual-task interference -- underscoring that blanket claims about LLM reasoning risk being both statistically premature and mechanistically misleading.
Abstract:The GSM-Symbolic benchmark (Mirzadeh et al., 2025) reported consistent performance drops across 25 Large Language Models (LLMs) when tested on template-generated variants of GSM8K problems, concluding that the models lack genuine reasoning capabilities. We argue that this conclusion rests on shaky statistical ground. Re-evaluating 20 open-weight models using bootstrapped Generalised Linear Mixed Models with per-question random effects, we find that only 8 exhibit statistically significant performance changes under the original prompt format. Moreover, we identify a previously unacknowledged factor: the distribution of integers in problem texts of the main GSM-Symbolic dataset is systematically shifted towards larger values relative to the original GSM8K (K-S statistic = 0.12, p < 0.001), contradicting the original authors' claims. Controlling for this large-number effect accounts for significance in half of the remaining cases. Among models with statistically significant performance deltas, we identify distinct, model-specific behavioural failure profiles -- including fragility of variable binding, arithmetic limitations, and dual-task interference -- underscoring that blanket claims about LLM reasoning risk being both statistically premature and mechanistically misleading.
41 pages, 11 figures, 23 tables. Accepted to EMNLP 2026 (main conference), track: Resources and Evaluation
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2605.28700 [cs.AI]
(or arXiv:2605.28700v3 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2605.28700 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 281 words
Related coverage
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · Sep 18, 2026
Who Gets to Define the Rules for AI?
Cohere · Sep 13, 2026
Inside the suddenly explosive world of AI safety
The Verge · Sep 17, 2026
A Key-Position Confound in no-cot-bench (and implications for interpreting looped transformers)
LessWrong · Oct 2, 2026