aiqbench analysis
This research identifies "rational value risk," a fundamental gap where even well-aligned large language models fail to maximize utility during complex reasoning tasks due to bounded prompts and imperfect verifiers. Experiments across major model families demonstrate that while alignment reduces this risk, it cannot eliminate it, though techniques like self-consistency and extended chains of thought offer partial mitigation with diminishing returns. The findings provide a rigorous mathematical framework for understanding the inherent limitations of current LLM architectures in executing optimal reasoning strategies.
In LLM Reasoning, there is Irrationality on top of Value Misalignment
arXiv:2606.20624v2 Announce Type: replace Abstract: Significant progress has been made in aligning LLMs with target value functions. We argue that, even when an LLM has been well aligned in (post-)training, it may still fail to maximise the aligned value in reasoning. We mathematically formalise this gap as rational value risk: the utility discrepancy between a model's deployed reasoning strategy and its rational counterpart whose responses maximise utility in the steepest direction. The estimation error of rational value risk is further decomposed into three components from bounded prompts, bounded responses, and imperfect verifiers. Extensive experiments are conducted, covering models Llama-3.1, Qwen-2.5, T\"ulu-3 families (7B-72B), GPT-5.2, GPT-5.5, and DeepSeek-V4, and benchmarks UltraFeedback, AlpacaEval, GSM8K, MATH, HumanEval, and MathArena. The results validate that (1) rational value risk is widespread; (2) value alignment can reduce, but cannot avoid, it; (3) self-consistency can improve rationality; and (4) a longer chain of thought improves rationality but with diminishing returns. The code is at https://github.com/EVIEHub/LLM-Rationality
Abstract:Significant progress has been made in aligning LLMs with target value functions. We argue that, even when an LLM has been well aligned in (post-)training, it may still fail to maximise the aligned value in reasoning. We mathematically formalise this gap as rational value risk: the utility discrepancy between a model's deployed reasoning strategy and its rational counterpart whose responses maximise utility in the steepest direction. The estimation error of rational value risk is further decomposed into three components from bounded prompts, bounded responses, and imperfect verifiers. Extensive experiments are conducted, covering models Llama-3.1, Qwen-2.5, Tülu-3 families (7B-72B), GPT-5.2, GPT-5.5, and DeepSeek-V4, and benchmarks UltraFeedback, AlpacaEval, GSM8K, MATH, HumanEval, and MathArena. The results validate that (1) rational value risk is widespread; (2) value alignment can reduce, but cannot avoid, it; (3) self-consistency can improve rationality; and (4) a longer chain of thought improves rationality but with diminishing returns. The code is at this https URL
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2606.20624 [cs.AI]
(or arXiv:2606.20624v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.20624 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 257 words
Related coverage
Problems of Proliferation
LessWrong · Sep 29, 2026
The Alignment Problem: Computer Science Meets Analytic Philosophy
AI Alignment Forum · Oct 18, 2026
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · Oct 16, 2026
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · Sep 23, 2026