analiza aiqbench
The paper proposes a way to regularize online reinforcement learning toward a pretrained diffusion or flow-matching base policy, penalizing actions by their distance from the base policy's support rather than by their divergence from its full distribution. Reported results on 19 tasks across four robotic benchmarks are a rise in average success rate from 33% to 95%, but these are the authors' own claims and depend on their benchmarks and baselines. For ML and robotics readers, the practical question is whether the sample-only distance estimate holds up in harder, real-world settings.
GSBR: Geometric Support Behavior Regularization for Online RL
arXiv:2611.05295v1 Announce Type: cross Abstract: Regularizing online reinforcement learning (RL) toward a pretrained base policy can improve sample efficiency and learning stability, but regularization that is too restrictive limits how much the policy can improve. Base policies -- typically diffusion or flow-matching models trained by imitation learning on large, diverse datasets -- capture many behaviors that are reasonable but suboptimal. Existing regularizers using such base policies are too restrictive in one of two ways: behavioral cloning regularizers pull the actor toward the full base-policy distribution, penalizing it for discarding suboptimal modes, while hard support constraints forbid improvements that lie just outside the base policy's support. We propose Geometric Support Behavior Regularization (GSBR), which penalizes actions by their distance from the base policy's support. Actions within the support incur no penalty, so the actor can concentrate on high-value modes; outside the support, where value estimates are less reliable, the penalty grows smoothly with distance while still permitting improvement when the value gained outweighs the penalty. GSBR estimates this distance from a finite set of base-policy samples, so it requires only sample access to the base policy. Across 19 tasks on four robotic benchmarks, GSBR raises the average success rate from 33% for the base policy to 95%, achieving the highest average success rate and throughput among all compared methods, with about 23% higher normalized throughput than the next-best RL method. Project website and open-source code at https://charlesxu0124.github.io/gsbr/ .
Abstract:Regularizing online reinforcement learning (RL) toward a pretrained base policy can improve sample efficiency and learning stability, but regularization that is too restrictive limits how much the policy can improve. Base policies -- typically diffusion or flow-matching models trained by imitation learning on large, diverse datasets -- capture many behaviors that are reasonable but suboptimal. Existing regularizers using such base policies are too restrictive in one of two ways: behavioral cloning regularizers pull the actor toward the full base-policy distribution, penalizing it for discarding suboptimal modes, while hard support constraints forbid improvements that lie just outside the base policy's support. We propose Geometric Support Behavior Regularization (GSBR), which penalizes actions by their distance from the base policy's support. Actions within the support incur no penalty, so the actor can concentrate on high-value modes; outside the support, where value estimates are less reliable, the penalty grows smoothly with distance while still permitting improvement when the value gained outweighs the penalty. GSBR estimates this distance from a finite set of base-policy samples, so it requires only sample access to the base policy. Across 19 tasks on four robotic benchmarks, GSBR raises the average success rate from 33% for the base policy to 95%, achieving the highest average success rate and throughput among all compared methods, with about 23% higher normalized throughput than the next-best RL method. Project website and open-source code at this https URL .
Robotics (cs.RO); Machine Learning (cs.LG)
arXiv:2611.05295 [cs.RO]
(or arXiv:2611.05295v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2611.05295 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 330 cuvinte
Subiecte conexe
CLONE: Continuous Latent Optimization for Normal Estimation via 3D Gaussian Splatting
arXiv · 5 aug. 2026
Isotropic Yet Undecodable: The Sequential Content-Sufficiency Gap in Latent-Predictive Text Representations
arXiv · 7 oct. 2026
Variational autoencoders with latent high-dimensional steady geometric flows for dynamics
arXiv · 17 iun. 2026
Diverse via bounded Agreement: Geometric Regularization for Multimodal Fusion
arXiv · 1 sept. 2026