aiqbench analysis
This research addresses a critical bottleneck in self-evolving reasoning systems where single-model uncertainty rewards collapse as solvers gain confidence, effectively starving the learning process. By introducing a multi-solver disagreement reward mechanism, the study enables curricula that target genuine capability boundaries rather than model-specific biases, leading to measurable improvements on complex math benchmarks. This approach offers a scalable, drop-in solution for enhancing robustness in autonomous AI training frameworks without requiring external human data.
Beyond Uncertainty: Multi-Solver Disagreement Rewards for Self-Evolving Reasoning Curricula
arXiv:2608.30035v1 Announce Type: new Abstract: Self-evolving reasoning frameworks train a Challenger to generate questions exposing a Solver's weaknesses, creating adaptive curricula without human data. However, existing approaches use a single solver's sampling uncertainty as the Challenger's reward. This creates a fundamental bottleneck: as the solver grows confident on the Challenger's question distribution, all sampled answers converge identically, collapsing the reward to zero and starving the Challenger of learning signal. Critically, this single-model reward cannot distinguish genuinely easy questions from those that merely align with one solver's learned biases. We propose a multi-solver disagreement reward using a heterogeneous ensemble varying in model capacity and sampling temperature. A normalized Shannon entropy over the ensemble's per-question plurality answers explicitly rewards questions where solvers produce conflicting solutions---capturing difficulty as inter-model divergence rather than intra-model sampling variance. This richer gradient enables the Challenger to discover questions targeting true capability boundaries, producing a curriculum that forces downstream Solvers to develop robust reasoning strategies generalizing across problem types. Our approach is a drop-in reward function replacement requiring no framework modifications or additional data. Experiments with Qwen3-4B show that Solvers trained on disagreement-Challenger questions achieve +1.34 points average improvement on competition-math benchmarks (MATH-500, AMC, Olympiad), suggesting that multi-solver disagreement provides a complementary and scalable signal for curriculum generation in self-play reasoning systems.
Abstract:Self-evolving reasoning frameworks train a Challenger to generate questions exposing a Solver's weaknesses, creating adaptive curricula without human data. However, existing approaches use a single solver's sampling uncertainty as the Challenger's reward. This creates a fundamental bottleneck: as the solver grows confident on the Challenger's question distribution, all sampled answers converge identically, collapsing the reward to zero and starving the Challenger of learning signal. Critically, this single-model reward cannot distinguish genuinely easy questions from those that merely align with one solver's learned biases. We propose a multi-solver disagreement reward using a heterogeneous ensemble varying in model capacity and sampling temperature. A normalized Shannon entropy over the ensemble's per-question plurality answers explicitly rewards questions where solvers produce conflicting solutions---capturing difficulty as inter-model divergence rather than intra-model sampling variance. This richer gradient enables the Challenger to discover questions targeting true capability boundaries, producing a curriculum that forces downstream Solvers to develop robust reasoning strategies generalizing across problem types. Our approach is a drop-in reward function replacement requiring no framework modifications or additional data. Experiments with Qwen3-4B show that Solvers trained on disagreement-Challenger questions achieve +1.34 points average improvement on competition-math benchmarks (MATH-500, AMC, Olympiad), suggesting that multi-solver disagreement provides a complementary and scalable signal for curriculum generation in self-play reasoning systems.
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2608.30035 [cs.AI]
(or arXiv:2608.30035v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.30035 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.1145/3799682.3840051 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 316 words
Related coverage
CoEvo: Oracle-Grounded Self-Evolution of a Single Model for Multi-Step Causal Reasoning
arXiv · Sep 23, 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026
Self-Evolve With a Reference:Anchored Training of Tool-Integrated Agents
arXiv · Oct 8, 2026