aiqbench analysis
This study clarifies the utility of Muon optimization in sparse-reward agentic reinforcement learning, demonstrating significant performance gains over AdamW when applied selectively to hidden weight matrices under specific policy optimizers like GiGPO and GRPO. The results highlight that benefits are highly dependent on hyperparameter choices such as learning rate and advantage estimator type rather than being universally applicable across all RL settings. These findings provide a crucial data point for researchers optimizing post-training pipelines for autonomous agents without relying solely on standard stochastic gradient descent methods.
When Does Muon Help Agentic Reinforcement Learning?
arXiv:2607.16169v2 Announce Type: replace-cross Abstract: Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning (RL) post-training remains unclear. We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%); high-rate AdamW controls retain no post-update success. The effect depends on the advantage estimator and learning rate. At 3e-5, Muon improves GRPO from 0.161 to 0.268, whereas GraphGPO's late-window gap narrows near saturation. At 1e-5, GraphGPO Muon reaches 0.901, raises normalized validation AUC from 0.399 to 0.556, and reaches 0.5 and 0.75 success 30 and 60 updates earlier, respectively. These exploratory results show that Muon can benefit agentic RL and motivate studying the policy optimizer, advantage estimator, and learning rate jointly.
Title:When Does Muon Help Agentic Reinforcement Learning?
Abstract:Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning (RL) post-training remains unclear. We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%); high-rate AdamW controls retain no post-update success. The effect depends on the advantage estimator and learning rate. At 3e-5, Muon improves GRPO from 0.161 to 0.268, whereas GraphGPO's late-window gap narrows near saturation. At 1e-5, GraphGPO Muon reaches 0.901, raises normalized validation AUC from 0.399 to 0.556, and reaches 0.5 and 0.75 success 30 and 60 updates earlier, respectively. These exploratory results show that Muon can benefit agentic RL and motivate studying the policy optimizer, advantage estimator, and learning rate jointly.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2607.16169 [cs.LG]
(or arXiv:2607.16169v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.16169 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 239 words
Related coverage
When Does Muon Help Agentic Reinforcement Learning?
arXiv · Jul 20, 2026
[AINews] OpenAI DevDay 2026: Dots, 6.1 Sol, Ultrafast, Decisions API, Agents API, Spaces, Marketplace, and 1.2 Billion ChatGPT WAU
Latent Space · Sep 30, 2026
Complete Guide to Thinking Machines Inkling
Analytics Vidhya · Jul 20, 2026
Dyna-2: General Purpose Physical Agents with Hour-Long Autonomy — Jason Ma, Dyna Robotics
Latent Space · Oct 13, 2026