aiqbench analysis
Reinforcement learning enhances LLM reasoning but introduces a non-monotonic relationship between output length and accuracy, peaking at an intermediate chain-of-thought length. While sample accuracy may plateau or decline, mode accuracy continues to improve with length, suggesting that dispersion around a correct center drives this behavior. This finding provides a critical data point for optimizing inference costs by identifying the precise reasoning length that maximizes performance without unnecessary computational overhead.
On the Optimal Reasoning Length for RL-Trained Language Models
arXiv:2602.09591v3 Announce Type: replace-cross Abstract: Reinforcement learning substantially improves reasoning in large language models, but it also tends to lengthen chain-of-thought outputs and increase computational cost. Although length-control methods have been proposed, the length-accuracy relationship they induce remains unclear. We train policies with several length-control methods on multiple base models in a controlled setup and find that, across both mathematical reasoning and code generation, accuracy is non-monotonic in output length, peaking at an intermediate value. Mode accuracy, however, continues to improve with length even in settings where sample accuracy plateaus or declines, indicating that the non-monotonic length-accuracy relationship is driven by dispersion around an increasingly correct center.
Abstract:Reinforcement learning substantially improves reasoning in large language models, but it also tends to lengthen chain-of-thought outputs and increase computational cost. Although length-control methods have been proposed, the length-accuracy relationship they induce remains unclear. We train policies with several length-control methods on multiple base models in a controlled setup and find that, across both mathematical reasoning and code generation, accuracy is non-monotonic in output length, peaking at an intermediate value. Mode accuracy, however, continues to improve with length even in settings where sample accuracy plateaus or declines, indicating that the non-monotonic length-accuracy relationship is driven by dispersion around an increasingly correct center.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2602.09591 [cs.CL]
(or arXiv:2602.09591v3 [cs.CL] for this version)
Full text under the publisher's open licence · 120 words