aiqbench analysis
This paper introduces PAC, a novel curriculum learning method for multi-task reinforcement learning in large language models that dynamically allocates training resources based on both policy update magnitude and actual reward gains. By integrating these signals via Bayesian Thompson Sampling, the approach significantly improves sample efficiency and final performance across diverse reasoning tasks compared to existing baselines. This advancement addresses a critical bottleneck in RL post-training by ensuring computational budgets are directed toward tasks that genuinely enhance model capabilities rather than those with merely large but ineffective updates.
PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs
arXiv:2608.30528v1 Announce Type: new Abstract: Reinforcement learning (RL) is used to improve the reasoning abilities of LLMs, while training data span heterogeneous tasks. However, most RL post-training pipelines rely on fixed or manually designed task mixtures, even though task usefulness changes as training progresses. Online curriculum methods often define learnability by update magnitude, ignoring whether the update translates into reward gains, which can misallocate rollout budget toward tasks with large but ineffective updates. We propose PAC, a Progress-Augmented Advantage Curriculum for multi-task RL of LLMs that combines two task-level signals: advantage-derived learnability, which measures the magnitude of the policy update a task can induce, and recent reward gains, which show whether those updates have improved task performance. A Bayesian Thompson Sampling controller uses these signals to allocate rollouts across tasks during GRPO training. We evaluate PAC under two settings: a multi-level reasoning setting and a multi-domain reasoning setting. PAC improves sample efficiency and final performance: it reaches comparable validation scores with fewer rollout steps and achieves higher final averages than random sampling and advantage-based curriculum baselines in both settings. These results show that jointly tracking advantage signals and actual reward gains yields an effective online curriculum for LLM post-training.
Abstract:Reinforcement learning (RL) is used to improve the reasoning abilities of LLMs, while training data span heterogeneous tasks. However, most RL post-training pipelines rely on fixed or manually designed task mixtures, even though task usefulness changes as training progresses. Online curriculum methods often define learnability by update magnitude, ignoring whether the update translates into reward gains, which can misallocate rollout budget toward tasks with large but ineffective updates. We propose PAC, a Progress-Augmented Advantage Curriculum for multi-task RL of LLMs that combines two task-level signals: advantage-derived learnability, which measures the magnitude of the policy update a task can induce, and recent reward gains, which show whether those updates have improved task performance. A Bayesian Thompson Sampling controller uses these signals to allocate rollouts across tasks during GRPO training. We evaluate PAC under two settings: a multi-level reasoning setting and a multi-domain reasoning setting. PAC improves sample efficiency and final performance: it reaches comparable validation scores with fewer rollout steps and achieves higher final averages than random sampling and advantage-based curriculum baselines in both settings. These results show that jointly tracking advantage signals and actual reward gains yields an effective online curriculum for LLM post-training.
Machine Learning (cs.LG)
arXiv:2608.30528 [cs.LG]
(or arXiv:2608.30528v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.30528 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 288 words
Related coverage
Reflection’s Beam model hits 501 billion parameters
Quantum Zeitgeist · Oct 5, 2026
Improving Cross-Problem Vehicle Routing with Locally Augmented Preferences and Representation Disentanglement
arXiv · Aug 26, 2026
Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction-Reality Gaps
arXiv · Oct 6, 2026
Digital States 2026: AI Moves From Experiment to Infrastructure
govtech.com · Sep 24, 2026