analiza aiqbench
This research introduces Cognitive Pairwise Training (CPT) to address the tendency of reinforcement learning models to overconfidently answer when their reasoning is flawed. By leveraging pairwise comparisons of reasoning traces, CPT helps large language models internalize a boundary between trustworthy and unreliable inference rather than merely memorizing refusal patterns. The method demonstrates improved accuracy and robustness across multiple model scales, offering a scalable path to better aligning model confidence with actual reasoning quality.
Enhancing LLM Metacognition via Cognitive Pairwise Training
arXiv:2606.00869v2 Announce Type: replace Abstract: Reinforcement learning with verifiable rewards (RLVR) has become central to LLM reasoning, but its outcome-level rewards can make models more willing to give confident answers when evidence or reasoning is unreliable. Existing SFT or RL methods mainly teach LLMs to refuse or express uncertainty at the response level, which can overfit abstention behavior rather than improve reasoning reliability. To address this limitation, we propose Cognitive Pairwise Training (CPT), a cognitive mid-training alignment stage that turns pairwise comparisons over reasoning traces into a reusable alignment signal. By learning to distinguish trustworthy from flawed reasoning, CPT encourages the model to internalize a reasoning-quality discrimination boundary rather than memorize surface refusal patterns. Across five model scales and three model families, CPT improves the reasoning--metacognition trade-off. At 14B, CPT+RL outperforms the standard SFT+RL pipeline by +2.2 math-average points and +5.6 abstention-F1 points. Further analyses show that CPT improves trace quality and exhibits strong robustness and scalability across evaluation and training settings. Code and models are released at https://github.com/Tsinghua-dhy/CPT.
Abstract:Reinforcement learning with verifiable rewards (RLVR) has become central to LLM reasoning, but its outcome-level rewards can make models more willing to give confident answers when evidence or reasoning is unreliable. Existing SFT or RL methods mainly teach LLMs to refuse or express uncertainty at the response level, which can overfit abstention behavior rather than improve reasoning reliability. To address this limitation, we propose Cognitive Pairwise Training (CPT), a cognitive mid-training alignment stage that turns pairwise comparisons over reasoning traces into a reusable alignment signal. By learning to distinguish trustworthy from flawed reasoning, CPT encourages the model to internalize a reasoning-quality discrimination boundary rather than memorize surface refusal patterns. Across five model scales and three model families, CPT improves the reasoning--metacognition trade-off. At 14B, CPT+RL outperforms the standard SFT+RL pipeline by +2.2 math-average points and +5.6 abstention-F1 points. Further analyses show that CPT improves trace quality and exhibits strong robustness and scalability across evaluation and training settings. Code and models are released at this https URL.
Machine Learning (cs.LG)
arXiv:2606.00869 [cs.LG]
(or arXiv:2606.00869v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.00869 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 258 cuvinte
Subiecte conexe
Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs
arXiv · 15 oct. 2026
Cognitive Enhancement: Rethinking the Necessity of Role-Playing for Large Language Models
arXiv · 1 oct. 2026
Self-Modeling Interventions Modulate Emergent Misalignment
LessWrong · 5 oct. 2026
Enhancing brain age estimation with structural MRI and synthesized cerebral blood volume maps
arXiv · 3 sept. 2026