analiza aiqbench
This study reveals a critical limitation in current large language models regarding probabilistic reasoning, showing they fail on counterintuitive problems despite high accuracy on standard tasks. The findings highlight significant vulnerabilities to token bias and misleading prompt suggestions, indicating these systems are not yet genuine reasoners for discrete probability distributions. For developers relying on LLMs for quantitative applications, this underscores the urgent need for specialized training or architectural changes before deployment in risk-sensitive environments.
How reliable are LLMs when it comes to playing dice?
arXiv:2606.07515v2 Announce Type: replace-cross Abstract: We investigate the probabilistic reasoning capabilities of large language models through a controlled benchmarking study on discrete probability problems. We constructed two datasets, respectively a set of standard exercises and a set of counterintuitive exercises, designed to trigger heuristic reasoning, and evaluated 8 state-of-the-art models, each tested with and without Chain-of-Thought prompting. Models achieve an average accuracy of 0.96 on standard problems but only 0.59 on counterintuitive ones. We further provide empirical evidence of token bias: performance drops by over 20% when canonical formulations are replaced by disguised variants. Embedding misleading suggestions in the prompt reduces performance by up to 34%, with no model proving immune. Taken together, the reported findings suggest that current LLMs are not yet genuine probabilistic reasoners, despite their success in advanced mathematical problems.
Title:How reliable are LLMs when it comes to playing dice?
Abstract:We investigate the probabilistic reasoning capabilities of large language models through a controlled benchmarking study on discrete probability problems. We constructed two datasets, respectively a set of standard exercises and a set of counterintuitive exercises, designed to trigger heuristic reasoning, and evaluated 8 state-of-the-art models, each tested with and without Chain-of-Thought prompting. Models achieve an average accuracy of 0.96 on standard problems but only 0.59 on counterintuitive ones. We further provide empirical evidence of token bias: performance drops by over 20% when canonical formulations are replaced by disguised variants. Embedding misleading suggestions in the prompt reduces performance by up to 34%, with no model proving immune. Taken together, the reported findings suggest that current LLMs are not yet genuine probabilistic reasoners, despite their success in advanced mathematical problems.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Probability (math.PR)
arXiv:2606.07515 [cs.CL]
(or arXiv:2606.07515v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.07515 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 239 cuvinte
Subiecte conexe
UC-PSRO: Utility-Conditioned Policy-Space Response Oracles with a Communication-Dropout Curriculum for Game-Theoretic Course-of-Action Generation in Adversarial Swarms
arXiv · 18 aug. 2026
TreeCCA: Canonical Correlation Analysis via Gradient-Boosted Trees
arXiv · 30 iul. 2026
Uncertainty-Aware Prototype Learning for Open-Set Test-Time Adaptation
arXiv · 5 nov. 2026
Are Two LLMs Better Than One? A Student-Teacher Dual-Head LLMs Architecture for Pharmaceutical Content Optimization
arXiv · 23 oct. 2026