analiza aiqbench
The paper studies whether a language model can improve its own reasoning principles for tracking who knows what in stories, and whether those learned principles are correct, not just effective. Its main finding is that a rule can raise benchmark accuracy while still failing on specific cases, so accuracy alone is not enough to validate a learned strategy. Readers working on LLM self-improvement or evaluation should note the proposed step of translating rules into code and testing them on synthetic cases with known answers.
CogRSI: Evolving and Auditing Cognitive Reasoning Strategies
arXiv:2611.01113v1 Announce Type: new Abstract: Can a language model learn better ways to reason about other people's minds by improving its own strategies, and are the strategies it learns actually right? We study this with cogrsi. A grounded reader marks who saw each event in a story and uses a short written principle to settle disagreements between its readings. With a principle written by experts, it beats self-consistency at the same token cost. A self-improvement loop then writes the principle itself. It keeps an update only if accuracy rises by a calibrated margin, and in one variant only if it also passes counterfactual test pairs. The learned principles transfer: on 800 BigToM pairs that no system had seen, each of them scores higher than the expert principle. We then check what the learned rules actually say. Each rule is translated into code and run on thousands of small stories with known answers. Most of them give a wrong answer in at least one situation they claim to cover, for example by treating a character who is in the room as knowing what happened even while asleep. Accuracy does not reveal which rules are wrong, and the counterfactual test does not fix this. A strategy can help in practice and still be wrong as a rule, so both need checking. Our code and data are publicly available.
Abstract:Can a language model learn better ways to reason about other people's minds by improving its own strategies, and are the strategies it learns actually right? We study this with cogrsi. A grounded reader marks who saw each event in a story and uses a short written principle to settle disagreements between its readings. With a principle written by experts, it beats self-consistency at the same token cost. A self-improvement loop then writes the principle itself. It keeps an update only if accuracy rises by a calibrated margin, and in one variant only if it also passes counterfactual test pairs. The learned principles transfer: on 800 BigToM pairs that no system had seen, each of them scores higher than the expert principle. We then check what the learned rules actually say. Each rule is translated into code and run on thousands of small stories with known answers. Most of them give a wrong answer in at least one situation they claim to cover, for example by treating a character who is in the room as knowing what happened even while asleep. Accuracy does not reveal which rules are wrong, and the counterfactual test does not fix this. A strategy can help in practice and still be wrong as a rule, so both need checking. Our code and data are publicly available.
Computation and Language (cs.CL)
arXiv:2611.01113 [cs.CL]
(or arXiv:2611.01113v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2611.01113 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 316 cuvinte
Subiecte conexe
Continuous-Utility Direct Preference Optimization
arXiv · 8 oct. 2026
Learning Explainable Representations of Complex Game-playing Strategies
arXiv · 7 oct. 2026
Autopilot Thinking
LessWrong · 7 iun. 2026
UCO: A Multi-Turn Interactive Reinforcement Learning Method for Adaptive Teaching with Large Language Models
arXiv · 26 aug. 2026