aiqbench analysis
This paper proposes a way to learn state-dependent discount factors in deep actor-critic RL, aimed at a known failure mode where the learned discount can game its own training target. The authors claim theoretical results for tabular MDPs and empirical gains on SAC and PPO, plus one industrial A/B test. Readers should weigh the tabular proofs and benchmark results as early evidence, since the deep-RL claims depend on implementation details and broader replication.
AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning
arXiv:2605.06149v2 Announce Type: replace-cross Abstract: The discount factor in reinforcement learning controls both the effective planning horizon and the strength of bootstrapping, yet most deep RL methods use a single fixed value across all states. State-dependent discounting is a natural way to adapt temporal credit assignment to local state structure, but naively learning a discount function inside deep actor--critic methods creates a self-referential target: the learned discount scales the same bootstrap term against which it is trained. Consequently, TD-error-based objectives can admit a degenerate shortcut, reducing target dependence on downstream value estimates instead of learning meaningful temporal structure. We propose \textbf{AdaGamma}, a practical framework for state-dependent discounting in deep actor--critic reinforcement learning. AdaGamma learns a bounded discount function \(\gamma_\phi(s)\) and trains it using a return-consistency objective that matches a one-step adaptive bootstrap to a multi-step return computed with a lagged, stop-gradient reference discount. This removes the direct target manipulation path and encourages the discount to encode state-dependent value propagation. We analyze the Bellman operator induced by a fixed state-dependent discount in finite tabular MDPs, proving contraction, exact soft policy iteration, and Lipschitz sensitivity under standard boundedness assumptions. Empirically, AdaGamma integrates into SAC and PPO, avoids TD-collapse, improves continuous-control benchmarks, and achieves statistically significant gains in a four-week online A/B test on the JD Logistics platform.
Abstract:The discount factor in reinforcement learning controls both the effective planning horizon and the strength of bootstrapping, yet most deep RL methods use a single fixed value across all states. State-dependent discounting is a natural way to adapt temporal credit assignment to local state structure, but naively learning a discount function inside deep actor--critic methods creates a self-referential target: the learned discount scales the same bootstrap term against which it is trained. Consequently, TD-error-based objectives can admit a degenerate shortcut, reducing target dependence on downstream value estimates instead of learning meaningful temporal structure. We propose \textbf{AdaGamma}, a practical framework for state-dependent discounting in deep actor--critic reinforcement learning. AdaGamma learns a bounded discount function \(\gamma_\phi(s)\) and trains it using a return-consistency objective that matches a one-step adaptive bootstrap to a multi-step return computed with a lagged, stop-gradient reference discount. This removes the direct target manipulation path and encourages the discount to encode state-dependent value propagation. We analyze the Bellman operator induced by a fixed state-dependent discount in finite tabular MDPs, proving contraction, exact soft policy iteration, and Lipschitz sensitivity under standard boundedness assumptions. Empirically, AdaGamma integrates into SAC and PPO, avoids TD-collapse, improves continuous-control benchmarks, and achieves statistically significant gains in a four-week online A/B test on the JD Logistics platform.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.06149 [cs.LG]
(or arXiv:2605.06149v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.06149 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 305 words
Related coverage
Thompson Sampling for Infinite-Horizon Discounted Decision Processes
arXiv · Sep 22, 2026
Information-Time Proximal Policy Optimization
arXiv · Sep 22, 2026
Q($\gamma_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning
arXiv · Oct 16, 2026
Active Inference as a Convex Markov Decision Process
arXiv · Jul 23, 2026