aiqbench analysis
This paper gives theoretical regret guarantees for value-based reinforcement learning on discretized controlled diffusions, with bounds that stay stable as the timestep shrinks. For ML readers, the key point is that the $\mathcal{O}(\log K)$ and $\mathcal{O}(\log K\log\log K)$ results hold under stated stability and regularity assumptions, and the constants do not blow up with finer discretization. Those assumptions are the main limits on applicability, so the work is best read as a formal analysis of a specific model-based fitted-Q algorithm rather than a general recipe for continuous-control RL.
Frequency-Uniform Log-Regret Reinforcement Learning for Diffusions
arXiv:2610.36758v1 Announce Type: cross Abstract: High-frequency control problems are often studied through time discretization, but refining the timestep $\delta$ increases the discrete horizon $H=T/\delta$ and can make standard finite-horizon regret guarantees deteriorate even when the physical horizon $T$ is fixed. We study episodic value-based reinforcement learning for Euler discretizations of controlled diffusions. The performance is measured as the cumulative regret over $K$ episodes relative to the optimal policy of the corresponding Euler model. Under stability and regularity conditions on value function gradients, Bellman residual derivatives, and fitted critics, we prove an $\mathcal{O}(\log K)$ regret bound with constants uniform in $\delta$. For dynamics with drift linear in unknown parameters and unknown constant diffusion covariance, we verify localized versions of these conditions for a constrained, model-based controlled-diffusion fitted-$Q$ algorithm (CD-FQ) and obtain high-probability $\mathcal{O}(\log K\log\log K)$ regret for fixed and finite $T$, again with constants uniform in $\delta$.
Abstract:High-frequency control problems are often studied through time discretization, but refining the timestep $\delta$ increases the discrete horizon $H=T/\delta$ and can make standard finite-horizon regret guarantees deteriorate even when the physical horizon $T$ is fixed. We study episodic value-based reinforcement learning for Euler discretizations of controlled diffusions. The performance is measured as the cumulative regret over $K$ episodes relative to the optimal policy of the corresponding Euler model. Under stability and regularity conditions on value function gradients, Bellman residual derivatives, and fitted critics, we prove an $\mathcal{O}(\log K)$ regret bound with constants uniform in $\delta$. For dynamics with drift linear in unknown parameters and unknown constant diffusion covariance, we verify localized versions of these conditions for a constrained, model-based controlled-diffusion fitted-$Q$ algorithm (CD-FQ) and obtain high-probability $\mathcal{O}(\log K\log\log K)$ regret for fixed and finite $T$, again with constants uniform in $\delta$.
Optimization and Control (math.OC); Machine Learning (cs.LG)
arXiv:2610.36758 [math.OC]
(or arXiv:2610.36758v1 [math.OC] for this version)
https://doi.org/10.48550/arXiv.2610.36758 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 239 words