analiza aiqbench
This paper introduces Expectile $n$-step Q-learning (ENQ), a novel off-policy reinforcement learning algorithm designed to mitigate the pessimistic bias inherent in multi-step returns caused by suboptimal logged actions. By replacing standard temporal-difference loss with an asymmetric expectile loss, ENQ achieves competitive performance across 27 manipulation and navigation tasks while demonstrating improved training throughput compared to existing methods like Long-Horizon Q-learning. Theoretical proofs confirm the operator's contraction properties under both deterministic and stochastic dynamics, offering a more robust approach for scaling RL agents in complex environments.
Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning
arXiv:2608.02034v1 Announce Type: new Abstract: Multi-step returns accelerate reward propagation in off-policy reinforcement learning, but couple the evaluation of each decision to the suboptimal logged actions that follow it, inducing a pessimistic bias that grows with the horizon. We propose Expectile $n$-step Q-learning (ENQ), which replaces the symmetric $n$-step temporal-difference (TD) loss with an asymmetric expectile loss on the action-value error, with expectile level $\tau$ as the only method-specific hyperparameter added beyond $n$-step TD. We prove that the ENQ operator is a $\gamma^{n}$-contraction. Under deterministic dynamics, at $\tau=1$, its bias vanishes at the optimal action-value function $Q^*$ on covered in-support pairs, and the corresponding fixed point satisfies the separation-$n$ instance and its multiples of the lower-bound inequality used by Long-Horizon Q-learning (LQL). Under stochastic dynamics, the operator bias admits two-sided bounds with horizon-independent noise constants. Using a single expectile level $\tau=0.8$ and a fixed backup horizon across 27 manipulation and navigation task instances, ENQ is competitive with LQL on aggregate, achieves higher measured training-step throughput in our profiling study, and benefits more from a ten-critic ensemble in a controlled scaling experiment.
Abstract:Multi-step returns accelerate reward propagation in off-policy reinforcement learning, but couple the evaluation of each decision to the suboptimal logged actions that follow it, inducing a pessimistic bias that grows with the horizon. We propose Expectile $n$-step Q-learning (ENQ), which replaces the symmetric $n$-step temporal-difference (TD) loss with an asymmetric expectile loss on the action-value error, with expectile level $\tau$ as the only method-specific hyperparameter added beyond $n$-step TD. We prove that the ENQ operator is a $\gamma^{n}$-contraction. Under deterministic dynamics, at $\tau=1$, its bias vanishes at the optimal action-value function $Q^*$ on covered in-support pairs, and the corresponding fixed point satisfies the separation-$n$ instance and its multiples of the lower-bound inequality used by Long-Horizon Q-learning (LQL). Under stochastic dynamics, the operator bias admits two-sided bounds with horizon-independent noise constants. Using a single expectile level $\tau=0.8$ and a fixed backup horizon across 27 manipulation and navigation task instances, ENQ is competitive with LQL on aggregate, achieves higher measured training-step throughput in our profiling study, and benefits more from a ten-critic ensemble in a controlled scaling experiment.
Machine Learning (cs.LG)
arXiv:2608.02034 [cs.LG]
(or arXiv:2608.02034v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.02034 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 270 cuvinte
Subiecte conexe
Scaling RL with verl on AMD Instinct MI355X: Async Walkthrough and Sync Benchmark
AMD · 18 aug. 2026
STRIDER: Stepping-Enabled Multi-Gait Hierarchical 3D Loco-Manipulation Framework for Humanoid Robots
arXiv · 22 sept. 2026
Adaptive Instructed-Retriever: Frontier-Quality Search at 2x Lower Latency
Databricks · 9 sept. 2026
TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Training
arXiv · 15 oct. 2026