aiqbench analysis
This study challenges the reliability of "unlearnable" prompts in Reinforcement Learning with Verifiable Rewards (RLVR), revealing that difficulty labels derived from limited sampling are highly unstable and can shift significantly across different experimental seeds. While the core phenomenon of slow learning persists, the authors demonstrate that much of the observed gradient separation is an artifact of insufficient correct rollouts rather than a fundamental barrier to optimization. For researchers in post-training reasoning, this work underscores the critical need for rigorous measurement protocols before declaring specific benchmarks as resistant to improvement.
Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVR
arXiv:2609.40115v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has become an important approach for improving reasoning during post-training. Recent work suggests that some difficult prompts remain resistant to learning even when they occasionally produce correct solutions. We revisit this unlearnability phenomenon and find that the affected prompts do improve, at roughly one third of the learnable rate, while the difficulty-defined set used to study them is much less reproducible than expected. These difficulty labels are estimated from a limited number of sampled responses. Combining them across seeds can further change which prompts are selected instead of simply reducing measurement noise. We develop a sampling-based framework for quantifying this instability and determining how much evaluation is required for difficulty assignments to reproduce reliably. We also revisit the gradient-similarity evidence proposed to explain unlearnability and show that part of the observed separation arises because difficult prompts provide fewer correct rollouts from which their gradients can be estimated. Matching this sample count weakens the gradient difference but does not remove it. Overall, the slow-learning phenomenon survives our reanalysis, while both the prompts used to define it and the evidence used to explain it require more careful measurement.
Title:Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVR
Abstract:Reinforcement learning with verifiable rewards (RLVR) has become an important approach for improving reasoning during post-training. Recent work suggests that some difficult prompts remain resistant to learning even when they occasionally produce correct solutions. We revisit this unlearnability phenomenon and find that the affected prompts do improve, at roughly one third of the learnable rate, while the difficulty-defined set used to study them is much less reproducible than expected. These difficulty labels are estimated from a limited number of sampled responses. Combining them across seeds can further change which prompts are selected instead of simply reducing measurement noise. We develop a sampling-based framework for quantifying this instability and determining how much evaluation is required for difficulty assignments to reproduce reliably. We also revisit the gradient-similarity evidence proposed to explain unlearnability and show that part of the observed separation arises because difficult prompts provide fewer correct rollouts from which their gradients can be estimated. Matching this sample count weakens the gradient difference but does not remove it. Overall, the slow-learning phenomenon survives our reanalysis, while both the prompts used to define it and the evidence used to explain it require more careful measurement.
Accepted at the NeurIPS 2026 Workshop on Transitioning from Pre-Training to Post-Training. Project page: this https URL
Artificial Intelligence (cs.AI)
arXiv:2609.40115 [cs.AI]
(or arXiv:2609.40115v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.40115 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 314 words
Related coverage
Unforgettable Generalization in Language Models
arXiv · Aug 6, 2026
CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL
arXiv · Oct 1, 2026
Simulated Users & Sad AIs
LessWrong · Jul 27, 2026
Preparing data for supervised fine-tuning Part 1: Formatting and quality
AWS · Aug 26, 2026