aiqbench analysis
This research investigates Reinforcement Learning with Verifiable Rewards (RLVR), revealing a critical trade-off where optimizing for high-reward outputs narrows the model's access to verified solutions rather than broadly expanding its reasoning capabilities. The study introduces "empirical support dynamics" to quantify this phenomenon, showing that current RLVR practices often increase reliability on known answers while suppressing diversity in token generation and potentially losing access to other correct solutions. These findings offer a necessary diagnostic for researchers aiming to balance accuracy with the preservation of a model's full solution space.
The Invisible Leash: Why RLVR May or May Not Escape Its Origin
arXiv:2507.14843v5 Announce Type: replace-cross Abstract: Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it remains unclear whether the current practice of RLVR truly expands models' reasoning boundary or mainly amplifies high-reward outputs that base models already know. We introduce empirical support dynamics, a framework that quantifies how RLVR preserves, expands, and shrinks access to verified solutions. Applied across models, reasoning tasks, and training checkpoints, it matches verified successes question by question to reveal changes that aggregate accuracy can conceal. Across the evaluated model-level benchmark aggregates, \textit{the shrinkage of observed support outweighs its expansion}: more questions lose a verified success than gain one at the stated sampling budgets. These losses occur alongside gains in \texttt{pass@1}. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an ``invisible leash'' in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.
Abstract:Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it remains unclear whether the current practice of RLVR truly expands models' reasoning boundary or mainly amplifies high-reward outputs that base models already know. We introduce empirical support dynamics, a framework that quantifies how RLVR preserves, expands, and shrinks access to verified solutions. Applied across models, reasoning tasks, and training checkpoints, it matches verified successes question by question to reveal changes that aggregate accuracy can conceal. Across the evaluated model-level benchmark aggregates, \textit{the shrinkage of observed support outweighs its expansion}: more questions lose a verified success than gain one at the stated sampling budgets. These losses occur alongside gains in \texttt{pass@1}. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an ``invisible leash'' in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2507.14843 [cs.LG]
(or arXiv:2507.14843v5 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2507.14843 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 289 words
Related coverage
Some Interesting Papers on RLVR
LessWrong · Jun 9, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
Practical NLP in the Browser with Transformers.js
KDnuggets · May 29, 2026
Agentic threat actor hits the orchestration plane: AI agent-driven container escape
Sysdig Blog · Jun 4, 2026