analiza aiqbench
This paper proposes a particle-based variational inference framework that addresses common RLHF issues like mode collapse and distribution drift within a Mixture-of-Experts architecture. By replacing fixed clipping or KL schedules with a geometry-based proximal-control mechanism, the method achieves significant performance gains, including a +179 ELO improvement on Codeforces and a 32% reduction in token count for mathematical reasoning. The approach offers a promising direction for stabilizing policy optimization in large sparse models without relying on traditional regularization schedules.
Variational Proximal Policy Optimization
arXiv:2606.08032v1 Announce Type: cross Abstract: Reinforcement Learning from Human Feedback via Proximal Policy Optimization often suffers from policy mode collapse, brittle exploration loops, and distribution drift. This paper introduces Variational Proximal Policy Optimization (\(\textsc{VP}_2\textsc{O}\)), a particle-based variational inference framework that maps policy optimization to Stein Variational Gradient Descent within a Mixture-of-Experts architecture. By leveraging functional kernels over localized expert prototypes alongside an expert orthogonalization loss, \(\textsc{VP}_2\textsc{O}\) introduces a geometry-based proximal-control mechanism that can reduce reliance on fixed clipping or KL schedules. Our results on a 33B/4B sparse Mixture-of-Experts model show several improvements across complex reasoning benchmarks, establishing a \(+\mathbf{179}\) ELO gain on Codeforces and a \(\mathbf{32\%}\) reduction in token count on AIME mathematical reasoning tasks.
Abstract:Reinforcement Learning from Human Feedback via Proximal Policy Optimization often suffers from policy mode collapse, brittle exploration loops, and distribution drift. This paper introduces Variational Proximal Policy Optimization (\(\textsc{VP}_2\textsc{O}\)), a particle-based variational inference framework that maps policy optimization to Stein Variational Gradient Descent within a Mixture-of-Experts architecture. By leveraging functional kernels over localized expert prototypes alongside an expert orthogonalization loss, \(\textsc{VP}_2\textsc{O}\) introduces a geometry-based proximal-control mechanism that can reduce reliance on fixed clipping or KL schedules. Our results on a 33B/4B sparse Mixture-of-Experts model show several improvements across complex reasoning benchmarks, establishing a \(+\mathbf{179}\) ELO gain on Codeforces and a \(\mathbf{32\%}\) reduction in token count on AIME mathematical reasoning tasks.
Machine Learning (stat.ML); Machine Learning (cs.LG)
arXiv:2606.08032 [stat.ML]
(or arXiv:2606.08032v1 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2606.08032 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 207 cuvinte
Subiecte conexe
Rethinking the Trust Region in LLM Reinforcement Learning
arXiv · 15 iun. 2026
Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning
arXiv · 6 oct. 2026
Debugging Logprob Mismatches in LLM Reinforcement Learning
AMD · 23 sept. 2026
P3: Probabilistic Policy Propagation for Stable VAE-Based Robot Learning
arXiv · 29 iul. 2026