aiqbench analysis
This research addresses a critical vulnerability in Reinforcement Learning with Verifiable Rewards (RLVR), where imperfect verifiers incentivize models to generate incorrect but accepted responses. The authors demonstrate that standard observation data is insufficient to detect these "reward hacking" behaviors and propose a correction mechanism using audit feedback to achieve selective control over error rates. By balancing verifier pressure against correctness, the method effectively reduces accepted errors while maintaining high response accuracy, offering a vital pathway for safer RL training pipelines.
Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control
arXiv:2609.35677v1 Announce Type: new Abstract: In reinforcement learning with verifiable rewards (RLVR), imperfect verifiers can reward incorrect responses, creating opportunities for reward hacking. Using gradient flow with a fixed verifier, we characterize the conditions under which reward rises while correctness falls. We then show that the observations available during RLVR are, in general, insufficient to detect or identify accepted errors, or to guarantee their reduction without sacrificing correct responses. To address this limit, we construct a correction using additional feedback about correctness from audits. This correction achieves \emph{selective control}: at the current policy, it lowers the probability of accepted errors and raises that of correct responses, provided it outweighs the pressure toward errors from verifier reward. Experiments with log linear and neural contextual bandits and with a language model support the analysis and show that selective control under partial auditing reduces accepted errors while increasing correctness.
Abstract:In reinforcement learning with verifiable rewards (RLVR), imperfect verifiers can reward incorrect responses, creating opportunities for reward hacking. Using gradient flow with a fixed verifier, we characterize the conditions under which reward rises while correctness falls. We then show that the observations available during RLVR are, in general, insufficient to detect or identify accepted errors, or to guarantee their reduction without sacrificing correct responses. To address this limit, we construct a correction using additional feedback about correctness from audits. This correction achieves \emph{selective control}: at the current policy, it lowers the probability of accepted errors and raises that of correct responses, provided it outweighs the pressure toward errors from verifier reward. Experiments with log linear and neural contextual bandits and with a language model support the analysis and show that selective control under partial auditing reduces accepted errors while increasing correctness.
Artificial Intelligence (cs.AI)
arXiv:2609.35677 [cs.AI]
(or arXiv:2609.35677v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.35677 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 235 words
Related coverage
Mastering Agentic Techniques: AI Agent Reinforcement Learning
NVIDIA · Jul 1, 2026
Why do models task game?
AI Alignment Forum · Aug 6, 2026
Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs
LessWrong · Jul 27, 2026
In open RLVR, “improvement” depends on the instrument — a small GRPO testbed separating what training optimizes, measures, and teaches
LessWrong · Jun 15, 2026