analiza aiqbench
This research introduces ACSAC, a reinforcement learning algorithm that dynamically adjusts action chunk sizes using a causal Transformer critic to solve long-horizon, sparse-reward tasks. By balancing reactivity with temporal consistency without task-specific tuning, the method demonstrates improved performance on manipulation and navigation benchmarks across offline and online settings. The work advances foundational RL techniques by addressing bootstrapping error accumulation through adaptive policy evaluation.
ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network
arXiv:2605.11009v2 Announce Type: replace Abstract: Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning, since single-step TD learning suffers from bootstrapping error accumulation across successive Bellman updates. Actor-critic methods with action chunking address this by operating over temporally extended actions, which reduce the effective horizon, enable fast value backups, and support temporally consistent exploration. However, existing methods rely on a fixed chunk size and therefore cannot adaptively balance reactivity against temporal consistency. A large fixed chunk size reduces responsiveness to new observations, while a small one produces incoherent motions, forcing task-specific tuning of the chunk size. To address this limitation, we propose Adaptive Chunk Size Actor-Critic (ACSAC). ACSAC leverages a causal Transformer critic to evaluate expected returns for action chunks of different sizes. At each chunk boundary, it adaptively selects the chunk size that maximizes the expected return, supporting flexible, state-dependent chunk sizes. We prove that the ACSAC Bellman operator is a $\gamma$-contraction whose unique fixed point is the action-value function of the adaptive policy. Experiments on OGBench demonstrate that ACSAC achieves strong performance on long-horizon, sparse-reward manipulation and navigation tasks across both offline RL and offline-to-online RL settings.
Abstract:Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning, since single-step TD learning suffers from bootstrapping error accumulation across successive Bellman updates. Actor-critic methods with action chunking address this by operating over temporally extended actions, which reduce the effective horizon, enable fast value backups, and support temporally consistent exploration. However, existing methods rely on a fixed chunk size and therefore cannot adaptively balance reactivity against temporal consistency. A large fixed chunk size reduces responsiveness to new observations, while a small one produces incoherent motions, forcing task-specific tuning of the chunk size. To address this limitation, we propose Adaptive Chunk Size Actor-Critic (ACSAC). ACSAC leverages a causal Transformer critic to evaluate expected returns for action chunks of different sizes. At each chunk boundary, it adaptively selects the chunk size that maximizes the expected return, supporting flexible, state-dependent chunk sizes. We prove that the ACSAC Bellman operator is a $\gamma$-contraction whose unique fixed point is the action-value function of the adaptive policy. Experiments on OGBench demonstrate that ACSAC achieves strong performance on long-horizon, sparse-reward manipulation and navigation tasks across both offline RL and offline-to-online RL settings.
Machine Learning (cs.LG); Robotics (cs.RO)
arXiv:2605.11009 [cs.LG]
(or arXiv:2605.11009v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.11009 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 279 cuvinte
Subiecte conexe
Facet-0: A Robotic Foundation Model for Contact-Rich Precise Manipulation
arXiv · 2 sept. 2026
Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
AWS · 4 sept. 2026
The Expensive-Loop Problem
LessWrong · 12 oct. 2026
Debugging Logprob Mismatches in LLM Reinforcement Learning
AMD · 23 sept. 2026