analiza aiqbench
This paper addresses a critical instability in visual reinforcement learning where standard architectural components like Batch Normalization cause policy mismatch and reward collapse during Proximal Policy Optimization (PPO) training. The authors propose Mode-Dependent Rectification (MDR), a lightweight dual-phase procedure that stabilizes on-policy optimization without requiring changes to the underlying network architecture. By resolving distributional drift between training and evaluation modes, this method offers a practical solution for improving the robustness of RL agents in complex environments.
Mode-Dependent Rectification for Stable PPO Training
arXiv:2602.05619v2 Announce Type: replace-cross Abstract: Mode-dependent architectural components (layers that behave differently during training and evaluation, such as Batch Normalization or dropout) are commonly used in visual reinforcement learning but can destabilize on-policy optimization. We show that in Proximal Policy Optimization (PPO), discrepancies between training and evaluation behavior induced by Batch Normalization lead to policy mismatch, distributional drift, and reward collapse. We propose Mode-Dependent Rectification (MDR), a lightweight dual-phase training procedure that stabilizes PPO under mode-dependent layers without architectural changes. Experiments across procedurally generated games and real-world patch-localization tasks demonstrate that MDR consistently improves stability and performance, and extends naturally to other mode-dependent layers.
Abstract:Mode-dependent architectural components (layers that behave differently during training and evaluation, such as Batch Normalization or dropout) are commonly used in visual reinforcement learning but can destabilize on-policy optimization. We show that in Proximal Policy Optimization (PPO), discrepancies between training and evaluation behavior induced by Batch Normalization lead to policy mismatch, distributional drift, and reward collapse. We propose Mode-Dependent Rectification (MDR), a lightweight dual-phase training procedure that stabilizes PPO under mode-dependent layers without architectural changes. Experiments across procedurally generated games and real-world patch-localization tasks demonstrate that MDR consistently improves stability and performance, and extends naturally to other mode-dependent layers.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2602.05619 [cs.LG]
(or arXiv:2602.05619v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.05619 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 194 cuvinte
Subiecte conexe
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026
Bring your own model with Amazon SageMaker AI: Script mode in SDK v3
AWS · 26 aug. 2026
In open RLVR, “improvement” depends on the instrument — a small GRPO testbed separating what training optimizes, measures, and teaches
LessWrong · 15 iun. 2026
[AINews] Muse Spark 1.3 matches GPT-5.6-Sol, confirming Meta Superintelligence as the newest Frontier Lab, >90% discount for training
Latent Space · 3 sept. 2026