analiza aiqbench
This paper is primarily about reinforcement learning theory: whether differential TD learning, an average-reward algorithm, stays convergent when its step sizes use a global clock rather than per-state visit counts. The authors show that the local-clock/global-clock equivalence known for discounted RL fails in the average-reward setting, with a counterexample, empirical divergence, and a proof that the global-clock ODE is unstable. For ML researchers, the practical takeaway is that step-size schedule choice can matter for stability in average-reward RL, so results proven for discounted settings should not be assumed to transfer.
On the Divergence of Differential Temporal Difference Learning without Local Clocks
arXiv:2605.06874v2 Announce Type: replace Abstract: Learning rate is a critical component of reinforcement learning (RL). This work uses global and local clocks to distinguish two types of learning rates. The former is of the standard form $\alpha_t$ that depends only on the time step $t$ (i.e., a global clock). The latter is of the form $\alpha_{\nu(S_t, t)}$, where $\nu(s, t)$ counts the number of visits to state $s$ until time $t$ (i.e., a local clock). In discounted RL, an RL algorithm that is convergent with a local clock is always also convergent with a global clock, and vice versa. We are not aware of any counterexample. The key contribution of this work is to show that this nice correspondence breaks down in average-reward RL. Specifically, we construct a counterexample showing that although differential temporal difference (TD) learning is convergent with a local clock, it can diverge with a global clock. We empirically observe this divergence in our experiments and also prove that the associated ODE of differential TD with a global clock is unstable. This counterexample closes the open problem in Wan et al. [2021], Blaser et al. [2026].
Abstract:Learning rate is a critical component of reinforcement learning (RL). This work uses global and local clocks to distinguish two types of learning rates. The former is of the standard form $\alpha_t$ that depends only on the time step $t$ (i.e., a global clock). The latter is of the form $\alpha_{\nu(S_t, t)}$, where $\nu(s, t)$ counts the number of visits to state $s$ until time $t$ (i.e., a local clock). In discounted RL, an RL algorithm that is convergent with a local clock is always also convergent with a global clock, and vice versa. We are not aware of any counterexample. The key contribution of this work is to show that this nice correspondence breaks down in average-reward RL. Specifically, we construct a counterexample showing that although differential temporal difference (TD) learning is convergent with a local clock, it can diverge with a global clock. We empirically observe this divergence in our experiments and also prove that the associated ODE of differential TD with a global clock is unstable. This counterexample closes the open problem in Wan et al. [2021], Blaser et al. [2026].
Machine Learning (cs.LG)
arXiv:2605.06874 [cs.LG]
(or arXiv:2605.06874v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.06874 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 276 cuvinte
Subiecte conexe
Task-aware Kernel Stein certificates for R\'{e}nyi differential privacy
arXiv · 19 oct. 2026
Federated and differentially private estimation of KL divergence
arXiv · 24 aug. 2026
Differentially Private Distribution Release of Gaussian Mixture Models via KL-Divergence Minimization
arXiv · 16 oct. 2026
$\nabla$Mimic: Ultra-Efficient Humanoid Control with Differentiable Simulation
arXiv · 2 nov. 2026