aiqbench analysis
This research addresses a critical failure mode in co-evolutionary self-play where language models converge on narrow problem distributions that stall learning. By introducing vocabulary dropout to mask output logits, the authors successfully maintain diversity in generated problems and significantly improve mathematical reasoning capabilities without human supervision. The technique effectively mimics game-theoretic constraints within pure language generation pipelines, offering a scalable path for unsupervised curriculum learning.
Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution
arXiv:2604.03472v4 Announce Type: replace-cross Abstract: Co-evolutionary self-play, where one language model generates problems and another solves them, promises curriculum learning without human supervision. The promise breaks down early in practice. The proposer converges to a narrow distribution of problems that satisfy the reward function, and the collapsed curriculum teaches the solver little, stalling the loop. We introduce vocabulary dropout, a lightweight intervention that randomly masks the proposer's output logits during both policy training and curriculum generation. The mask is hard and non-stationary, so the proposer cannot lock into fixed token sequences. Training Qwen3-4B and Qwen3-8B on mathematical reasoning via R-Zero, vocabulary dropout sustains proposer diversity throughout training across lexical, semantic, and functional measures, and improves the solver by an average of +4.4 points at 8B with the largest gains on competition-level benchmarks. Explicit action-space constraints, filling the structural role that game rules fill in classical self-play, can keep co-evolution in language productive. Vocabulary dropout is one simple way to impose them.
Abstract:Co-evolutionary self-play, where one language model generates problems and another solves them, promises curriculum learning without human supervision. The promise breaks down early in practice. The proposer converges to a narrow distribution of problems that satisfy the reward function, and the collapsed curriculum teaches the solver little, stalling the loop. We introduce vocabulary dropout, a lightweight intervention that randomly masks the proposer's output logits during both policy training and curriculum generation. The mask is hard and non-stationary, so the proposer cannot lock into fixed token sequences. Training Qwen3-4B and Qwen3-8B on mathematical reasoning via R-Zero, vocabulary dropout sustains proposer diversity throughout training across lexical, semantic, and functional measures, and improves the solver by an average of +4.4 points at 8B with the largest gains on competition-level benchmarks. Explicit action-space constraints, filling the structural role that game rules fill in classical self-play, can keep co-evolution in language productive. Vocabulary dropout is one simple way to impose them.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2604.03472 [cs.CL]
(or arXiv:2604.03472v4 [cs.CL] for this version)
Full text under the publisher's open licence · 171 words
Related coverage
Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution
arXiv · Jun 17, 2026
Dr. Zero: Self-Evolving Search Agents without Training Data
arXiv · Jul 23, 2026
The art and science of hyperparameter optimization on Amazon Nova Forge
AWS · Jun 2, 2026
From Foundational Competency to Expert Performance: A Curriculum Approach to Model Development
Snorkel AI Blog · Sep 15, 2026