aiqbench analysis
This research addresses a critical bottleneck in autonomous curriculum learning where co-evolving models tend to converge on narrow problem distributions. By applying vocabulary dropout as an explicit action-space constraint, the study successfully maintains diversity in generated problems and significantly improves solver performance on mathematical reasoning benchmarks. The technique demonstrates that lightweight structural interventions can effectively prevent reward hacking without requiring human supervision or heavy computational overhead.
Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution
arXiv:2604.03472v3 Announce Type: replace Abstract: Co-evolutionary self-play, where one language model generates problems and another solves them, promises autonomous curriculum learning without human supervision. In practice, the proposer quickly converges to a narrow distribution of problems that satisfy the reward function. This diversity collapse renders the curriculum uninformative for the solver, stalling the co-evolutionary loop. We introduce vocabulary dropout, a random mask applied to the proposer's output logits during both policy training and curriculum generation, as a lightweight mechanism to sustain diversity. The mask is hard and non-stationary, preventing the proposer from locking into fixed token sequences. Training Qwen3-4B and Qwen3-8B on mathematical reasoning via R-Zero, we find that vocabulary dropout sustains proposer diversity across lexical, semantic, and functional metrics throughout training. It also yields solver improvements averaging +4.4 points at 8B, with the largest gains on competition-level benchmarks. Our findings suggest that explicit action-space constraints, analogous to the structural role that game rules play in classical self-play, can help sustain productive co-evolution in language. Vocabulary dropout is one simple instantiation of this principle.
Abstract:Co-evolutionary self-play, where one language model generates problems and another solves them, promises autonomous curriculum learning without human supervision. In practice, the proposer quickly converges to a narrow distribution of problems that satisfy the reward function. This diversity collapse renders the curriculum uninformative for the solver, stalling the co-evolutionary loop. We introduce vocabulary dropout, a random mask applied to the proposer's output logits during both policy training and curriculum generation, as a lightweight mechanism to sustain diversity. The mask is hard and non-stationary, preventing the proposer from locking into fixed token sequences. Training Qwen3-4B and Qwen3-8B on mathematical reasoning via R-Zero, we find that vocabulary dropout sustains proposer diversity across lexical, semantic, and functional metrics throughout training. It also yields solver improvements averaging +4.4 points at 8B, with the largest gains on competition-level benchmarks. Our findings suggest that explicit action-space constraints, analogous to the structural role that game rules play in classical self-play, can help sustain productive co-evolution in language. Vocabulary dropout is one simple instantiation of this principle.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2604.03472 [cs.CL]
(or arXiv:2604.03472v3 [cs.CL] for this version)
Full text under the publisher's open licence · 184 words
Related coverage
Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution
arXiv · Jul 23, 2026
Dr. Zero: Self-Evolving Search Agents without Training Data
arXiv · Jul 23, 2026
The art and science of hyperparameter optimization on Amazon Nova Forge
AWS · Jun 2, 2026
From Foundational Competency to Expert Performance: A Curriculum Approach to Model Development
Snorkel AI Blog · Sep 15, 2026