analiza aiqbench
This research addresses the challenge of sparse rewards in offline goal-conditioned reinforcement learning by introducing Reward Stimulation Implicit Q-Learning (RSIQL). The method improves training supervision by applying additional reward signals to intermediate states that demonstrate progress toward a goal, avoiding the complexity of hierarchical subgoal policies. Experiments on standard benchmarks confirm that this approach matches the performance of more complex hierarchical methods while maintaining a simpler flat policy structure.
Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulation
arXiv:2609.19414v1 Announce Type: new Abstract: Goal-conditioned reinforcement learning aims to learn policies that reach specified goals, but remains challenging in offline settings with sparse rewards and long-horizon dependencies. In such settings, goal-completion information can be temporally distant from the early decisions that enable success, while offline value estimation introduces additional error. We study this issue from a reward-propagation perspective and show, in a stylized delayed-goal setting, how goal-directed value separation can become small relative to local estimation error. Motivated by this analysis, we propose Reward Stimulation Implicit Q-Learning (RSIQL), a simple non-hierarchical method that introduces additional reward signals at progress-making intermediate states in offline trajectories. RSIQL uses an auxiliary goal-conditioned value function to identify intermediate states estimated to make progress toward the goal and applies reward stimulation to provide less-delayed training supervision. Unlike hierarchical methods, RSIQL does not learn a separate high-level subgoal policy. Experiments on D4RL goal-reaching benchmarks and OGBench show that RSIQL improves over goal-conditioned IQL on average and achieves performance competitive with hierarchical offline goal-conditioned methods, while retaining a simple flat policy structure.
Abstract:Goal-conditioned reinforcement learning aims to learn policies that reach specified goals, but remains challenging in offline settings with sparse rewards and long-horizon dependencies. In such settings, goal-completion information can be temporally distant from the early decisions that enable success, while offline value estimation introduces additional error. We study this issue from a reward-propagation perspective and show, in a stylized delayed-goal setting, how goal-directed value separation can become small relative to local estimation error. Motivated by this analysis, we propose Reward Stimulation Implicit Q-Learning (RSIQL), a simple non-hierarchical method that introduces additional reward signals at progress-making intermediate states in offline trajectories. RSIQL uses an auxiliary goal-conditioned value function to identify intermediate states estimated to make progress toward the goal and applies reward stimulation to provide less-delayed training supervision. Unlike hierarchical methods, RSIQL does not learn a separate high-level subgoal policy. Experiments on D4RL goal-reaching benchmarks and OGBench show that RSIQL improves over goal-conditioned IQL on average and achieves performance competitive with hierarchical offline goal-conditioned methods, while retaining a simple flat policy structure.
Machine Learning (cs.LG)
arXiv:2609.19414 [cs.LG]
(or arXiv:2609.19414v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.19414 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 265 cuvinte
Subiecte conexe
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · 12 iun. 2026
The Alignment Problem: Computer Science Meets Analytic Philosophy
LessWrong · 18 oct. 2026
MedPAIR: Measuring Whether Physicians and AI Agree on What Matters in Medical QA
Snorkel AI Blog · 2 oct. 2026
Offline Policy Evaluation as a decision support tool for designing Adaptive Experiments
arXiv · 28 sept. 2026