analiza aiqbench
This research establishes a fundamental performance ceiling for inference-time scaling, demonstrating that non-reasoning models hit an accuracy plateau regardless of increased compute usage. The authors further reveal that simple majority voting outperforms complex agent frameworks and identify specific linguistic markers as efficient proxies for response quality without additional computation. These findings redefine the efficiency limits of test-time compute strategies in large language model deployment.
Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals
arXiv:2504.14047v2 Announce Type: replace Abstract: There is intense interest in investigating how inference time compute (ITC) (e.g. repeated sampling, refinements, etc) can improve large language model (LLM) capabilities. While breakthroughs like DeepSeek-R1 highlight the power of reinforcement learning for reasoning, the interaction between ITC and reasoning-optimized weights remains poorly understood. This work conducts a comprehensive analysis of inference-time scaling methods for both reasoning and non-reasoning models on challenging reasoning tasks. While prior work suggests that scaling test-time compute can optimally substitute for model parameter scaling, we identify a fundamental limit to this compute-equivalence, the reasoning floor, a performance plateau that non-reasoning models cannot escape, no matter how much inference compute is spent. We demonstrate that general-purpose models fail to match the accuracy of reasoning-optimized models even with an order of magnitude more inference compute, suggesting that internalizing reasoning protocols is a prerequisite for effective test-time scaling. Within reasoning models, we find that the complexity of the scaling method often yields diminishing returns; simple majority voting consistently outperforms sophisticated sequential revision and mixture-of-agents frameworks. Crucially, we identify a Linguistic Signal of Correctness - correct responses are significantly more concise and exhibit a lower density of hedging and thinking markers. We demonstrate that these intrinsic linguistic features can serve as zero-compute proxies for response quality, providing a pathway to more efficient, self-diagnostic reasoning agents.
Abstract:There is intense interest in investigating how inference time compute (ITC) (e.g. repeated sampling, refinements, etc) can improve large language model (LLM) capabilities. While breakthroughs like DeepSeek-R1 highlight the power of reinforcement learning for reasoning, the interaction between ITC and reasoning-optimized weights remains poorly understood. This work conducts a comprehensive analysis of inference-time scaling methods for both reasoning and non-reasoning models on challenging reasoning tasks. While prior work suggests that scaling test-time compute can optimally substitute for model parameter scaling, we identify a fundamental limit to this compute-equivalence, the reasoning floor, a performance plateau that non-reasoning models cannot escape, no matter how much inference compute is spent. We demonstrate that general-purpose models fail to match the accuracy of reasoning-optimized models even with an order of magnitude more inference compute, suggesting that internalizing reasoning protocols is a prerequisite for effective test-time scaling. Within reasoning models, we find that the complexity of the scaling method often yields diminishing returns; simple majority voting consistently outperforms sophisticated sequential revision and mixture-of-agents frameworks. Crucially, we identify a Linguistic Signal of Correctness - correct responses are significantly more concise and exhibit a lower density of hedging and thinking markers. We demonstrate that these intrinsic linguistic features can serve as zero-compute proxies for response quality, providing a pathway to more efficient, self-diagnostic reasoning agents.
Artificial Intelligence (cs.AI)
arXiv:2504.14047 [cs.AI]
(or arXiv:2504.14047v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2504.14047 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 309 cuvinte
Subiecte conexe
The hard physics and complex economics of AI’s insatiable hunger for power
SiliconANGLE · 11 sept. 2026
Automate model quota request and operational issue triage on Amazon Bedrock
AWS · 3 iun. 2026
Humanity’s Last Invention — Richard Socher of Recursive
Latent Space · 14 sept. 2026
Productionizing TurboQuant on AMD GPUs for KV-Cache-Bound LLM Inference
AMD · 11 iun. 2026