Test-Time Scaling in Reasoning Models Is Not Effective for Knowledge-Intensive Tasks Yet

arXiv:2509.06861v3 Announce Type: replace-cross Abstract: Test-time scaling increases inference-time computation through longer reasoning chains and has shown strong performance gains across many domains. However, frontier models still suffer from factuality hallucinations, raising the question of whether increased computation is effective on closed-book knowledge-intensive tasks. In this work, we evaluate 14 reasoning models under different test-time scaling strategies. Our results challenge its effectiveness: increasing test-time computation does not consistently improve accuracy and often leads to more hallucinations. We find that changes in hallucination rates are largely driven by the model's willingness to answer, as longer reasoning encourages more attempts, many of which are incorrect. We also observe patterns consistent with confirmation bias, where extended reasoning reinforces early incorrect beliefs with fabricated details. Finally, we provide an information-theoretic perspective showing that compute-only test-time scaling, as a post-processing procedure of a fixed model, cannot introduce new information about the ground-truth answer, explaining the limited performance gains. Overall, our findings highlight important limitations of current test-time scaling methods for closed-book knowledge-intensive tasks. Code and data are available at https://github.com/XuZhao0/tts-knowledge

Abstract:Test-time scaling increases inference-time computation through longer reasoning chains and has shown strong performance gains across many domains. However, frontier models still suffer from factuality hallucinations, raising the question of whether increased computation is effective on closed-book knowledge-intensive tasks. In this work, we evaluate 14 reasoning models under different test-time scaling strategies. Our results challenge its effectiveness: increasing test-time computation does not consistently improve accuracy and often leads to more hallucinations. We find that changes in hallucination rates are largely driven by the model's willingness to answer, as longer reasoning encourages more attempts, many of which are incorrect. We also observe patterns consistent with confirmation bias, where extended reasoning reinforces early incorrect beliefs with fabricated details. Finally, we provide an information-theoretic perspective showing that compute-only test-time scaling, as a post-processing procedure of a fixed model, cannot introduce new information about the ground-truth answer, explaining the limited performance gains. Overall, our findings highlight important limitations of current test-time scaling methods for closed-book knowledge-intensive tasks. Code and data are available at this https URL

COLM 2026. 10+27 pages, 9 figures, 11 tables

Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)

arXiv:2509.06861 [cs.AI]

(or arXiv:2509.06861v3 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2509.06861 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

Test-Time Scaling in Reasoning Models Is Not Effective for Knowledge-Intensive Tasks Yet

Subiecte conexe