aiqbench analysis
This arXiv paper targets a concrete training problem: reinforcement learning from a model's own confidence (self-certainty) can drift toward overconfident but less accurate reasoning. SIF's gains are reported on 12 math and science benchmarks across three backbones, with the largest margin over Intuitor at 4.71 points on Qwen2.5-Math-1.5B. Readers should note these are preprint results on specific models and benchmarks, so the practical value depends on whether the selective-update approach holds up at larger scales and outside math-heavy evaluations.
Learning to Reason with Selective Internal Feedback
arXiv:2610.21392v1 Announce Type: new Abstract: Reinforcement learning from internal feedback can improve language models without ground-truth answers or external verifiers. However, confidence and correctness can diverge during training, and continued optimization of average self-certainty can increase confidence while degrading reasoning accuracy. We introduce SIF, a reinforcement learning method that combines comparative reward construction with selective policy updates. SIF compares ranked token-level self-certainty profiles across multiple windows and peers, assigning positive rewards only when both levels of voting provide sufficient support. Sample and group level filtering determines which responses contribute to policy optimization. Experiments on three backbones across 12 mathematical and scientific reasoning benchmarks show that SIF's final checkpoints achieve higher average accuracy than the best-validation checkpoints of Intuitor and Vigor, with a gain of 4.71 percentage points over Intuitor on Qwen2.5-Math-1.5B. Further analyses show better retention of accuracy during continued optimization, while test-time training experiments demonstrate sustained improvements with as few as four rollouts per prompt.
Abstract:Reinforcement learning from internal feedback can improve language models without ground-truth answers or external verifiers. However, confidence and correctness can diverge during training, and continued optimization of average self-certainty can increase confidence while degrading reasoning accuracy. We introduce SIF, a reinforcement learning method that combines comparative reward construction with selective policy updates. SIF compares ranked token-level self-certainty profiles across multiple windows and peers, assigning positive rewards only when both levels of voting provide sufficient support. Sample and group level filtering determines which responses contribute to policy optimization. Experiments on three backbones across 12 mathematical and scientific reasoning benchmarks show that SIF's final checkpoints achieve higher average accuracy than the best-validation checkpoints of Intuitor and Vigor, with a gain of 4.71 percentage points over Intuitor on Qwen2.5-Math-1.5B. Further analyses show better retention of accuracy during continued optimization, while test-time training experiments demonstrate sustained improvements with as few as four rollouts per prompt.
Artificial Intelligence (cs.AI)
arXiv:2610.21392 [cs.AI]
(or arXiv:2610.21392v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.21392 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 245 words
Related coverage
A Candid Abacus AI Review: The All-in-One AI Platform for Professionals & Enterprises
KDnuggets · Sep 10, 2026
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · Jun 12, 2026
Models Can't Remember Their Training. Neither Can You.
LessWrong · Jul 19, 2026
Why I Left Google DeepMind
LessWrong · Jul 15, 2026