aiqbench analysis
This research introduces S2L-PO, a framework that leverages smaller models within the same family as structured explorers to enhance Group Relative Policy Optimization (GRPO) without relying on noisy token-level randomness. By utilizing the inherent policy-level diversity of smaller models, the approach achieves faster convergence and higher performance ceilings on mathematical reasoning benchmarks while reducing overall compute costs. The method effectively balances exploration and exploitation through a progressive annealing strategy, offering a practical optimization path for training larger language models.
Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO
arXiv:2605.30789v2 Announce Type: replace-cross Abstract: We identify a new dimension for enhancing rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs. While GRPO relies on diverse rollouts, prevailing strategies primarily increase diversity by injecting more token-level randomness, which may introduce step-wise noise and lead to incoherent trajectories. We uncover that smaller models within the same model family inherently exhibit higher policy-level diversity, indicated by their superior pass@k relative to larger counterparts as sample counts increase. Unlike token-level noise, this diversity is temporally correlated, preserves logical consistency, and provides structured exploration signals for gradient estimation. We thus propose S2L-PO (Small-to-Large Policy Optimization), a framework that leverages fixed small models as natural explorers to train larger models. To balance exploration and exploitation, we design a progressive annealing strategy that transitions from offline small-model rollouts to the large learner's own sampling. This shift elegantly avoids mid-training performance drops caused by the small model's capacity limits, achieving faster convergence and unlocking a higher performance ceiling. S2L-PO improves accuracy on diverse mathematical reasoning benchmarks (e.g., +8.8% on AIME 24 using a 1.7B explorer to guide the 8B model) while reducing rollout compute.
Abstract:We identify a new dimension for enhancing rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs. While GRPO relies on diverse rollouts, prevailing strategies primarily increase diversity by injecting more token-level randomness, which may introduce step-wise noise and lead to incoherent trajectories. We uncover that smaller models within the same model family inherently exhibit higher policy-level diversity, indicated by their superior pass@k relative to larger counterparts as sample counts increase. Unlike token-level noise, this diversity is temporally correlated, preserves logical consistency, and provides structured exploration signals for gradient estimation. We thus propose S2L-PO (Small-to-Large Policy Optimization), a framework that leverages fixed small models as natural explorers to train larger models. To balance exploration and exploitation, we design a progressive annealing strategy that transitions from offline small-model rollouts to the large learner's own sampling. This shift elegantly avoids mid-training performance drops caused by the small model's capacity limits, achieving faster convergence and unlocking a higher performance ceiling. S2L-PO improves accuracy on diverse mathematical reasoning benchmarks (e.g., +8.8% on AIME 24 using a 1.7B explorer to guide the 8B model) while reducing rollout compute.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.30789 [cs.LG]
(or arXiv:2605.30789v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.30789 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 278 words
Related coverage
Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO
arXiv · Jul 28, 2026
Frontier post-training recipe review with Finbarr Timbers
Interconnects · Jun 16, 2026
Mastering Agentic Techniques: AI Agent Reinforcement Learning
NVIDIA · Jul 1, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026