aiqbench analysis
ThunderSyncRL introduces a novel training paradigm that eliminates pipeline inefficiencies in agentic reinforcement learning by streaming gradients without inducing policy staleness. By overlapping rollout and learning phases, the method achieves up to 1.9x faster training speeds on benchmarks like SWE-bench Verified while maintaining identical convergence properties to synchronous approaches. This advancement directly addresses the computational bottlenecks inherent in post-training large language models for complex, long-horizon tasks.
ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learning
arXiv:2610.05935v1 Announce Type: cross Abstract: Language models are moving beyond generating answers to pursuing long-horizon goals in interactive environments. Post-training these agents requires long, heterogeneous trajectories, and synchronous systems leave learner engines idle until rollout and verification finish. To squeeze out these pipeline bubbles, asynchronous training overlaps rollout and learning across updates, but comes at the cost of policy staleness. We introduce ThunderSyncRL, which starts gradient computation as soon as all required inputs are fixed, without policy staleness. For group relative policy optimization (GRPO), ThunderSyncRL computes each trajectory's score gradient as soon as the reward for that trajectory arrives, without waiting for the group. For on-policy distillation (OPD), it computes gradients for each completed agentic turn's teacher-scored actions while tool calls run in the sandbox. We prove that gradient streaming produces the same GRPO and OPD updates as batch-synchronous training, without changing either objective. On SWE-bench Verified and Terminal Bench 4.0, we train models to the same performance up to $1.9 \times$ faster than synchronous training. With zero policy staleness, ThunderSyncRL also outperforms asynchronous training at a fixed budget by up to $2.47$ percentage points.
Abstract:Language models are moving beyond generating answers to pursuing long-horizon goals in interactive environments. Post-training these agents requires long, heterogeneous trajectories, and synchronous systems leave learner engines idle until rollout and verification finish. To squeeze out these pipeline bubbles, asynchronous training overlaps rollout and learning across updates, but comes at the cost of policy staleness. We introduce ThunderSyncRL, which starts gradient computation as soon as all required inputs are fixed, without policy staleness. For group relative policy optimization (GRPO), ThunderSyncRL computes each trajectory's score gradient as soon as the reward for that trajectory arrives, without waiting for the group. For on-policy distillation (OPD), it computes gradients for each completed agentic turn's teacher-scored actions while tool calls run in the sandbox. We prove that gradient streaming produces the same GRPO and OPD updates as batch-synchronous training, without changing either objective. On SWE-bench Verified and Terminal Bench 4.0, we train models to the same performance up to $1.9 \times$ faster than synchronous training. With zero policy staleness, ThunderSyncRL also outperforms asynchronous training at a fixed budget by up to $2.47$ percentage points.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.05935 [cs.LG]
(or arXiv:2610.05935v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.05935 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 277 words
Related coverage
ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation
arXiv · Oct 14, 2026
News
Quantum Computing Report · Jul 25, 2026
Quantum Machines and Academia Sinica speed up qubit tuning with AI
Quantum Zeitgeist · Aug 30, 2026
AI and Quantum Computing Briefing
livetradingnews.com · Aug 4, 2026