aiqbench analysis
This research introduces Fast-Slow Thinking Reward Models (F/S-RM), a hybrid architecture that dynamically balances the efficiency of scalar judgments with the accuracy of chain-of-thought reasoning to optimize RLHF alignment. By leveraging a dual-confidence activation mechanism, the model achieves state-of-the-art benchmark scores while reducing token consumption by 22.5%, offering a practical solution for scaling reward modeling without prohibitive computational costs.
Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models
arXiv:2603.20212v2 Announce Type: replace-cross Abstract: Reward models (RMs) are critical for aligning Large Language Models via Reinforcement Learning from Human Feedback (RLHF). While Generative Reward Models (GRMs) achieve superior accuracy through chain-of-thought (CoT) reasoning, they incur substantial computational costs. Conversely, Scalar Reward Models (SRMs) offer efficiency but suffer from limited performance and adaptability in complex scenarios. We introduce Fast-Slow Thinking Reward Models (F/S-RM), a hybrid RM architecture inspired by Dual Process Theory. It trains a single model to integrate two distinct reward paradigms: scalar-style first-token pairwise judgment (fast thinking) and CoT-based judgment (slow thinking), regulated by a dual-confidence activation mechanism that determines when to activate slow thinking. Under hybrid inference, F/S-RM achieves state-of-the-art accuracy with an average score of 84.3 across benchmarks, while reducing token consumption by 22.5%.
Abstract:Reward models (RMs) are critical for aligning Large Language Models via Reinforcement Learning from Human Feedback (RLHF). While Generative Reward Models (GRMs) achieve superior accuracy through chain-of-thought (CoT) reasoning, they incur substantial computational costs. Conversely, Scalar Reward Models (SRMs) offer efficiency but suffer from limited performance and adaptability in complex scenarios. We introduce Fast-Slow Thinking Reward Models (F/S-RM), a hybrid RM architecture inspired by Dual Process Theory. It trains a single model to integrate two distinct reward paradigms: scalar-style first-token pairwise judgment (fast thinking) and CoT-based judgment (slow thinking), regulated by a dual-confidence activation mechanism that determines when to activate slow thinking. Under hybrid inference, F/S-RM achieves state-of-the-art accuracy with an average score of 84.3 across benchmarks, while reducing token consumption by 22.5%.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2603.20212 [cs.CL]
(or arXiv:2603.20212v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2603.20212 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 219 words
Related coverage
Humanity’s Last Invention — Richard Socher of Recursive
Latent Space · Sep 14, 2026
Presentation: Engineering AI for Creativity and Curiosity on Mobile
InfoQ · Jul 21, 2026
Microsoft’s AI chief says superintelligence is near, but won’t take your job
The Verge · Jun 8, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026