aiqbench analysis
This paper targets a concrete training bottleneck in reinforcement learning with verifiable rewards: as training progresses, many prompts become too easy (pass rate of 1), shrinking the useful data. Composition-RL addresses this by offline composition of existing verified prompts, which avoids changing the RL optimizer. The reported gains are on reasoning benchmarks across 4B to 30B models, and the results are tied to this specific setup, so readers should check them against their own data and compute budgets before generalizing.
Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
arXiv:2602.12036v3 Announce Type: replace Abstract: Large-scale verifiable prompts underpin the success of Reinforcement Learning with Verifiable Rewards (RLVR), but they contain many uninformative examples and are costly to expand further. Recent studies focus on better exploiting limited training data by prioritizing hard prompts whose rollout pass rate is 0. However, easy prompts with a pass rate of 1 also become increasingly prevalent as training progresses, thereby reducing the effective data size. To mitigate this, we propose Composition-RL, an offline data-augmentation approach that composes existing prompts before RL to address the pass-rate-1 bottleneck without modifying the RL optimizer. The composed dataset is generated and verified once from the fixed source pool and then used for RL training. Extensive experiments across model sizes from 4B to 30B show that Composition-RL consistently improves reasoning capability over RL trained on the original dataset. Performance can be further boosted with a curriculum variant of Composition-RL that gradually increases compositional depth over training. Additionally, Composition-RL enables more effective cross-domain RL by composing prompts drawn from different domains. All our code, data, and models are publicly available to facilitate future research.
Abstract:Large-scale verifiable prompts underpin the success of Reinforcement Learning with Verifiable Rewards (RLVR), but they contain many uninformative examples and are costly to expand further. Recent studies focus on better exploiting limited training data by prioritizing hard prompts whose rollout pass rate is 0. However, easy prompts with a pass rate of 1 also become increasingly prevalent as training progresses, thereby reducing the effective data size. To mitigate this, we propose Composition-RL, an offline data-augmentation approach that composes existing prompts before RL to address the pass-rate-1 bottleneck without modifying the RL optimizer. The composed dataset is generated and verified once from the fixed source pool and then used for RL training. Extensive experiments across model sizes from 4B to 30B show that Composition-RL consistently improves reasoning capability over RL trained on the original dataset. Performance can be further boosted with a curriculum variant of Composition-RL that gradually increases compositional depth over training. Additionally, Composition-RL enables more effective cross-domain RL by composing prompts drawn from different domains. All our code, data, and models are publicly available to facilitate future research.
NeurIPS 2026 camera-ready version. 33 pages
Computation and Language (cs.CL)
arXiv:2602.12036 [cs.CL]
(or arXiv:2602.12036v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2602.12036 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 278 words
Related coverage
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · Jun 12, 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · Sep 29, 2026
When Partial Credit Opposes Completion
Snorkel AI Blog · Oct 15, 2026
Evaluating task vectors, unlearning and inoculation
LessWrong · Sep 20, 2026