aiqbench analysis
This paper studies label-free reinforcement learning for LLMs, where a model improves using only its own outputs as training signals. It identifies "selection-induced diversity collapse," in which repeated self-selection narrows the set of correct solutions a model can sample. The proposed EVOL-RL method combines majority voting with embedding-based novelty to counter this, and reports gains in pass@n on math benchmarks. These are preprint results on small models and specific benchmarks, so readers should check whether the gains hold up under broader evaluation.
Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation
arXiv:2509.15194v4 Announce Type: replace Abstract: Large language models (LLMs) can improve through reinforcement learning with verifiable rewards (RLVR), but many settings lack reference answers or external verifiers. Label-free self-improvement instead constructs training signals from the model's own outputs. Because the same policy generates both the candidate responses and the signals derived from them, each update also changes the candidates available for future training, creating a self-reinforcing feedback loop. We identify a failure mode of this loop, which we call selection-induced diversity collapse: repeated self-selection can progressively narrow the candidate pool and make alternative correct solutions harder to sample. To address this problem, we propose EVOL-RL, which balances selection and variation. Majority voting provides a stable selection signal, while embedding-based novelty refines credit among responses with the same selection status; entropy regularization and asymmetric clipping further support this process by maintaining diverse rollouts and preserving rare, high-advantage updates. Across multiple training distributions, model scales, backbones, and matched label-free baselines, EVOL-RL consistently improves pass@n while generally maintaining or improving pass@1. For example, when trained without labels on AIME24, Qwen3-4B-Base improves over TTRL on held-out AIME25 from 4.6% to 17.1% in pass@1 and from 18.5% to 42.0% in pass@16. Training analyses further show that when the current majority is wrong, EVOL-RL better preserves alternative correct solutions. The code is available at: https://github.com/YujunZhou/EVOL-RL.
Abstract:Large language models (LLMs) can improve through reinforcement learning with verifiable rewards (RLVR), but many settings lack reference answers or external verifiers. Label-free self-improvement instead constructs training signals from the model's own outputs. Because the same policy generates both the candidate responses and the signals derived from them, each update also changes the candidates available for future training, creating a self-reinforcing feedback loop. We identify a failure mode of this loop, which we call selection-induced diversity collapse: repeated self-selection can progressively narrow the candidate pool and make alternative correct solutions harder to sample. To address this problem, we propose EVOL-RL, which balances selection and variation. Majority voting provides a stable selection signal, while embedding-based novelty refines credit among responses with the same selection status; entropy regularization and asymmetric clipping further support this process by maintaining diverse rollouts and preserving rare, high-advantage updates. Across multiple training distributions, model scales, backbones, and matched label-free baselines, EVOL-RL consistently improves pass@n while generally maintaining or improving pass@1. For example, when trained without labels on AIME24, Qwen3-4B-Base improves over TTRL on held-out AIME25 from 4.6% to 17.1% in pass@1 and from 18.5% to 42.0% in pass@16. Training analyses further show that when the current majority is wrong, EVOL-RL better preserves alternative correct solutions. The code is available at: this https URL.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2509.15194 [cs.LG]
(or arXiv:2509.15194v4 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2509.15194 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 312 words
Related coverage
Hamiltonian Expressibility for Ansatz Selection in Variational Quantum Algorithms
arXiv · Jul 23, 2026
RNA Design via Conditioned Flow Matching and Finite-Policy Reinforcement Learning
arXiv · Sep 30, 2026
AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning
arXiv · Oct 5, 2026
Variational objectives for amortized Bayesian inference in inverse problems: The role of posterior conditioning
arXiv · Sep 23, 2026