analiza aiqbench
This arXiv preprint studies how extra "thinking" time changes the way reasoning models interpret everyday language. Holding weights fixed, the authors find that longer reasoning tends to make models read "some" as "not all" and "or" as "not both," treating logical words strictly, while context-driven inferences such as indirect answers improve. Their trace analysis suggests models first consider the enriched reading and then cancel it, and an early-answer intervention is a tentative fix. For AI readers, this means test-time compute settings can shift interpretive behavior, not just accuracy, so evaluations should account for it.
Entertain, then Cancel: How Thinking Makes Reasoning Models Read Some and Or Literally
arXiv:2610.21204v1 Announce Type: new Abstract: When a speaker says ``some students passed'', listeners infer that not all of them did. Reasoning models now let users decide how long a model thinks before it answers, but we do not know what extra thinking does to such everyday inferences. We study this directly: we keep a model's weights fixed, change only how much it thinks, and record which reading it chooses. A second tool cuts the model's own reasoning at successive points and asks for an answer, showing where the reading changes. Thinking makes models read logical words as logic: it cancels the implicatures of \emph{some} (\emph{not all}) and \emph{or} (\emph{not both}) in most tested open reasoning models and in GPT-5, while context-driven inferences such as indirect answers improve. The trace shows why: models first entertain the enriched reading and cancel it later. Answering from that early reading, an exploratory inference-time remedy, recovers these implicatures without losing literal accuracy. Evaluations of reasoning models should measure how they interpret, not only how accurate they are.
Abstract:When a speaker says ``some students passed'', listeners infer that not all of them did. Reasoning models now let users decide how long a model thinks before it answers, but we do not know what extra thinking does to such everyday inferences. We study this directly: we keep a model's weights fixed, change only how much it thinks, and record which reading it chooses. A second tool cuts the model's own reasoning at successive points and asks for an answer, showing where the reading changes. Thinking makes models read logical words as logic: it cancels the implicatures of \emph{some} (\emph{not all}) and \emph{or} (\emph{not both}) in most tested open reasoning models and in GPT-5, while context-driven inferences such as indirect answers improve. The trace shows why: models first entertain the enriched reading and cancel it later. Answering from that early reading, an exploratory inference-time remedy, recovers these implicatures without losing literal accuracy. Evaluations of reasoning models should measure how they interpret, not only how accurate they are.
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2610.21204 [cs.AI]
(or arXiv:2610.21204v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.21204 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 265 cuvinte
