aiqbench analysis
This paper gives finite-time convergence bounds for actor-critic reinforcement learning that uses momentum-based actor updates (including Muon and normalized SGD variants) trained on a single Markovian trajectory. The rates are O(K^-1/3) for normalized steps and O(K^-1/4) for unnormalized steps, up to a distribution-mismatch term, so the guarantees are theoretical and the bounds may be loose in practice. For practitioners, the useful part is the comparison of optimizer geometry: it suggests when orthogonalized updates like Muon should beat normalized gradient steps, depending on gradient rank and curvature.
A Unified Analysis of Importance-Corrected Momentum Actor-Critic over a Markovian Trajectory
arXiv:2610.33477v1 Announce Type: new Abstract: We present a generic actor-critic method that combines a ridge-regularized pathwise least-squares temporal-difference (LSTD) critic with importance-corrected momentum (ICM) actors. The critic solves a ridge-regularized system per frame by Newton-Schulz inversion and needs no stepsize, no knowledge of the mixing time, and no rank condition. Averaging stochastic policy gradients over frames of $T$ consecutive transitions, the ICM actors incur a frame-dependent gradient bias of order $\frac{1}{T}$. To our knowledge, we establish the first finite-time bounds for ICM actor-critic methods with normalized and unnormalized steps along a single Markovian trajectory. Up to a stationary-to-discounted distribution mismatch, the averaged gradient norm over $K$ updates is bounded by ${\cal O}(K^{-1/3}+\frac{1}{T})$ for normalized steps and by ${\cal O}(K^{-1/4}+\frac{1}{T})$ for unnormalized steps. We specialize these bounds to Muon, normalized stochastic gradient descent (SGD) with momentum, SGD with momentum, and AdaNorm, and compare them in the nuclear norm. With a dimension-free stepsize, Muon confines the factor $\min\{m,n\}^{1/2}$ associated with an $m\times n$ actor parameter matrix to the noise and correction terms. The comparison also characterizes when orthogonalization guarantees a larger noiseless ascent than normalization: exactly when the gradient's nuclear rank exceeds the spectral-to-Frobenius curvature ratio.
Abstract:We present a generic actor-critic method that combines a ridge-regularized pathwise least-squares temporal-difference (LSTD) critic with importance-corrected momentum (ICM) actors. The critic solves a ridge-regularized system per frame by Newton-Schulz inversion and needs no stepsize, no knowledge of the mixing time, and no rank condition. Averaging stochastic policy gradients over frames of $T$ consecutive transitions, the ICM actors incur a frame-dependent gradient bias of order $\frac{1}{T}$. To our knowledge, we establish the first finite-time bounds for ICM actor-critic methods with normalized and unnormalized steps along a single Markovian trajectory. Up to a stationary-to-discounted distribution mismatch, the averaged gradient norm over $K$ updates is bounded by ${\cal O}(K^{-1/3}+\frac{1}{T})$ for normalized steps and by ${\cal O}(K^{-1/4}+\frac{1}{T})$ for unnormalized steps. We specialize these bounds to Muon, normalized stochastic gradient descent (SGD) with momentum, SGD with momentum, and AdaNorm, and compare them in the nuclear norm. With a dimension-free stepsize, Muon confines the factor $\min\{m,n\}^{1/2}$ associated with an $m\times n$ actor parameter matrix to the noise and correction terms. The comparison also characterizes when orthogonalization guarantees a larger noiseless ascent than normalization: exactly when the gradient's nuclear rank exceeds the spectral-to-Frobenius curvature ratio.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.33477 [cs.LG]
(or arXiv:2610.33477v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.33477 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 286 words