analiza aiqbench
This paper introduces GeZo-SAC, a novel reinforcement learning algorithm that employs geometric zonotope representations to dynamically adjust critic pessimism in off-policy actor-critic methods. By adapting the aggregation rule based on the disagreement between critics, the method achieves superior performance and efficiency on standard MuJoCo locomotion benchmarks while minimizing overestimation bias. The approach represents a significant methodological advancement in handling value function estimation within deep reinforcement learning frameworks.
A Geometric Approach to Soft Actor-Critic with Zonotopes for Locomotion Learning
arXiv:2610.12113v1 Announce Type: cross Abstract: Off-policy actor--critic methods control overestimation bias by taking the minimum of two critics. This uses the same aggregation rule everywhere, regardless of how the critics disagree. We propose \textbf{GeZo-SAC}, which uses auxiliary geometric representations to adapt critic pessimism to the state and action. Alongside its scalar value, each critic predicts a set of generators defining a zonotope. Probing this zonotope along sampled directions provides a geometric width, "subtracted from each critic value as a pessimistic offset, and a measure of disagreement between the two critics, aggregated with log-sum-exp. This disagreement controls how the critics are combined, moving from a width-weighted average toward the usual minimum as disagreement increases. At inference, the deployed policy is an unmodified SAC actor, since the generators are used only on the critic side during training.Across four MuJoCo-v5 locomotion benchmarks and six off-policy baselines, GeZo-SAC achieves the highest mean return on Ant-v5 and Hopper-v5 and remains competitive with other methods on the remaining tasks. Our analysis further shows that GeZo-SAC achieves the lowest average actuator work and action effort per metre among the evaluated methods, while maintaining near-zero measured overestimation frequency across all four environments.
Abstract:Off-policy actor--critic methods control overestimation bias by taking the minimum of two critics. This uses the same aggregation rule everywhere, regardless of how the critics disagree. We propose \textbf{GeZo-SAC}, which uses auxiliary geometric representations to adapt critic pessimism to the state and action. Alongside its scalar value, each critic predicts a set of generators defining a zonotope. Probing this zonotope along sampled directions provides a geometric width, "subtracted from each critic value as a pessimistic offset, and a measure of disagreement between the two critics, aggregated with log-sum-exp. This disagreement controls how the critics are combined, moving from a width-weighted average toward the usual minimum as disagreement increases. At inference, the deployed policy is an unmodified SAC actor, since the generators are used only on the critic side during this http URL four MuJoCo-v5 locomotion benchmarks and six off-policy baselines, GeZo-SAC achieves the highest mean return on Ant-v5 and Hopper-v5 and remains competitive with other methods on the remaining tasks. Our analysis further shows that GeZo-SAC achieves the lowest average actuator work and action effort per metre among the evaluated methods, while maintaining near-zero measured overestimation frequency across all four environments.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.12113 [cs.LG]
(or arXiv:2610.12113v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.12113 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 288 cuvinte
Subiecte conexe
Tractable Reinforcement Learning for Full Class of Signal Temporal Logic Specifications Using Spatiotemporal Tube Reward
arXiv · 24 sept. 2026
Problems of Proliferation
LessWrong · 29 sept. 2026
Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns
arXiv · 8 iun. 2026
MorFiC: Fixing Value Miscalibration for Zero-Shot Quadruped Transfer
arXiv · 25 aug. 2026