aiqbench analysis
This paper proposes a reinforcement learning method that trains several Q-learning heads, each with a different discount factor, and blends them at decision time based on the policy's entropy-derived confidence. The reported gains are concentrated on tasks where rewards are delayed, and the authors frame the approach as a lightweight add-on that works with standard value-learning methods. Readers should look for the benchmark details, the baselines, and the cost of maintaining multiple heads before judging how much it improves on tuning a single discount factor.
Q($\gamma_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning
arXiv:2610.21764v1 Announce Type: new Abstract: A fixed discount factor in Q-learning imposes a single planning horizon throughout training, despite the differing demands of early exploration and long-term credit assignment. We propose Q($\gamma_H$), a multi-horizon Q-learning framework that maintains multiple discount-specific heads and adaptively combines them at control time using an entropy-derived confidence signal. The resulting state-dependent mixing weights shift control from shorter- to longer-horizon heads as policy confidence increases, enabling adaptive horizon mixing without modifying individual Bellman updates. Across classical control and deep reinforcement-learning benchmarks, Q($\gamma_H$) shows its strongest improvements on delayed-credit tasks relative to strong fixed-discount baselines. By decoupling horizon adaptation from the underlying learning updates, Q($\gamma_H$) provides a lightweight framework for adaptive planning-horizon control in reinforcement learning, avoiding the need for manual discount-factor selection while remaining compatible with standard value-learning methods.
Abstract:A fixed discount factor in Q-learning imposes a single planning horizon throughout training, despite the differing demands of early exploration and long-term credit assignment. We propose Q($\gamma_H$), a multi-horizon Q-learning framework that maintains multiple discount-specific heads and adaptively combines them at control time using an entropy-derived confidence signal. The resulting state-dependent mixing weights shift control from shorter- to longer-horizon heads as policy confidence increases, enabling adaptive horizon mixing without modifying individual Bellman updates. Across classical control and deep reinforcement-learning benchmarks, Q($\gamma_H$) shows its strongest improvements on delayed-credit tasks relative to strong fixed-discount baselines. By decoupling horizon adaptation from the underlying learning updates, Q($\gamma_H$) provides a lightweight framework for adaptive planning-horizon control in reinforcement learning, avoiding the need for manual discount-factor selection while remaining compatible with standard value-learning methods.
15 pages. Accepted at ICONIP 2026
Machine Learning (cs.LG)
arXiv:2610.21764 [cs.LG]
(or arXiv:2610.21764v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.21764 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 228 words
Related coverage
[AINews] Claude Opus 5.5, the new default model for AINews — and everybody cuts prices 40-50%
Latent Space · Sep 23, 2026
Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation
arXiv · Sep 16, 2026
Optimal Sample Complexity of Stable Discounted Markov Decision Processes
arXiv · Sep 22, 2026
Sharp Statistical Rates for Asynchronous TD Learning with Markovian Data
arXiv · Oct 1, 2026