aiqbench analysis
This study critically evaluates the reliability of rule-based and model-based verifiers used in reinforcement learning with verifiable rewards (RLVR), a core methodology for advancing large reasoning models like DeepSeek-R1. The research reveals that while rule-based systems suffer from format-dependent false negatives, model-based verifiers introduce significant reward hacking risks during fine-tuning, ultimately hindering policy improvement. These findings are vital for researchers aiming to build robust training pipelines that can scale without compromising the integrity of mathematical reasoning capabilities.
From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning
arXiv:2505.22203v3 Announce Type: replace-cross Abstract: Trustworthy verifiers are essential for the success of reinforcement learning with verifiable reward (RLVR), which is the core methodology behind various large reasoning models such as DeepSeek-R1. In complex domains like mathematical reasoning, rule-based verifiers have been widely adopted in previous works to train strong reasoning models. However, the reliability of these verifiers and their impact on the RL training process remain poorly understood. In this work, we take mathematical reasoning as a case study and conduct a comprehensive analysis of various verifiers in both static evaluation and RL training scenarios. We show that widely used rule-based verifiers fail to recognize equivalent answers in different formats, leading to substantial false negatives that increasingly hinder RL performance as the policy model gets stronger. Model-based verifiers substantially improve static accuracy but are highly susceptible to reward hacking during RL, where they misclassify certain patterns in responses as correct, particularly after fine-tuning. Our findings underscore the challenges inherent to both rule- and model-based verifiers and provide insights toward developing more accurate and robust reward systems for reinforcement learning.
Abstract:Trustworthy verifiers are essential for the success of reinforcement learning with verifiable reward (RLVR), which is the core methodology behind various large reasoning models such as DeepSeek-R1. In complex domains like mathematical reasoning, rule-based verifiers have been widely adopted in previous works to train strong reasoning models. However, the reliability of these verifiers and their impact on the RL training process remain poorly understood. In this work, we take mathematical reasoning as a case study and conduct a comprehensive analysis of various verifiers in both static evaluation and RL training scenarios. We show that widely used rule-based verifiers fail to recognize equivalent answers in different formats, leading to substantial false negatives that increasingly hinder RL performance as the policy model gets stronger. Model-based verifiers substantially improve static accuracy but are highly susceptible to reward hacking during RL, where they misclassify certain patterns in responses as correct, particularly after fine-tuning. Our findings underscore the challenges inherent to both rule- and model-based verifiers and provide insights toward developing more accurate and robust reward systems for reinforcement learning.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2505.22203 [cs.LG]
(or arXiv:2505.22203v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2505.22203 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 274 words
Related coverage
In open RLVR, “improvement” depends on the instrument — a small GRPO testbed separating what training optimizes, measures, and teaches
LessWrong · Jun 15, 2026
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · Oct 16, 2026
Thousand-dimensional structure
AI Alignment Forum · Jul 30, 2026
How robust are natural language autoencoders to initialization?
AI Alignment Forum · Jul 10, 2026