aiqbench analysis
The study finds that turning on reasoning makes different language models more likely to agree on the same inputs, but longer reasoning traces on a given input predict less consistent predictions across models. The distinction between reasoning as an intervention and reasoning as an observed behavior matters for anyone deploying multiple models or auditing them for consistency. Results come from nine models and three classification tasks, so they are a useful signal rather than a settled general law.
Does Reasoning Make Language Models Agree?
arXiv:2610.41691v1 Announce Type: new Abstract: Language models face two opposing concerns: monoculture, in which models err in correlated ways, and multiplicity, in which equally accurate models make conflicting decisions. The rise of reasoning models has complicated this debate: On the one hand, reasoning may increase model agreement if models anchor on the same salient features or follow similar reasoning paths. On the other hand, stochastic reasoning traces may introduce additional branch points leading to more diverse predictions. We resolve this tension by distinguishing reasoning as an intervention from reasoning as an observed behavior. We compare agreement under controlled changes in reasoning effort with agreement across observed reasoning lengths, measuring both against the agreement expected from model accuracy. Across nine models from two developers and three classification tasks, enabling reasoning increases agreement between models evaluated on the same individuals. Yet the individuals who elicit longer reasoning traces receive less consistent predictions across models. Both patterns persist after accounting for differences in accuracy. Asking models to reason more thus promotes agreement, while observing them reason longer signals disagreement.
Title:Does Reasoning Make Language Models Agree?
Abstract:Language models face two opposing concerns: monoculture, in which models err in correlated ways, and multiplicity, in which equally accurate models make conflicting decisions. The rise of reasoning models has complicated this debate: On the one hand, reasoning may increase model agreement if models anchor on the same salient features or follow similar reasoning paths. On the other hand, stochastic reasoning traces may introduce additional branch points leading to more diverse predictions. We resolve this tension by distinguishing reasoning as an intervention from reasoning as an observed behavior. We compare agreement under controlled changes in reasoning effort with agreement across observed reasoning lengths, measuring both against the agreement expected from model accuracy. Across nine models from two developers and three classification tasks, enabling reasoning increases agreement between models evaluated on the same individuals. Yet the individuals who elicit longer reasoning traces receive less consistent predictions across models. Both patterns persist after accounting for differences in accuracy. Asking models to reason more thus promotes agreement, while observing them reason longer signals disagreement.
Machine Learning (cs.LG); Computation and Language (cs.CL); Computers and Society (cs.CY)
arXiv:2610.41691 [cs.LG]
(or arXiv:2610.41691v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.41691 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 279 words
Related coverage
Multilingual Bridges: How Data Mixing Unlocks In-Language Reasoning
Cohere · Oct 6, 2026
Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoning
arXiv · Sep 10, 2026
Fine-tuning GPT-OSS-20B for multilingual reasoning on Amazon SageMaker HyperPod
AWS · Oct 20, 2026
Efficient Multilingual Reasoning Transfer via Progressive Code-Switching
arXiv · Jul 28, 2026