aiqbench analysis
This research addresses critical instability in diffusion policy reinforcement learning by solving the "double-drift" problem where surrogate objectives diverge from true likelihoods. The proposed DiPOD framework stabilizes training through self-distillation and on-policy regularization, enabling more reliable improvements for both language models and continuous control tasks. This advancement is essential for scaling robust generative agents in complex environments without sacrificing convergence guarantees.
Diffusion Policy Optimization without Drifting Apart
arXiv:2606.13795v1 Announce Type: new Abstract: RL post-training has become increasingly pivotal for improving diffusion policies, but existing diffusion policy-gradient methods are often unstable and cannot achieve reliable policy improvement. We identify the cause as the double-drift phenomenon: optimizing a variational surrogate can let the ELBO separate from the true log-likelihood, which then makes the resulting proxy policy gradient misaligned with the true policy gradient of expected return. We propose \textbf{DiPOD}, a diffusion policy optimization framework that maintains tight-bound behavior throughout training by interleaving self-distillation with policy-improving gradient updates. This leads to a simple and practical algorithm: augmenting each diffusion policy-gradient update with an on-policy ELBO regularizer. Across diffusion language model post-training and continuous-control diffusion policies, DiPOD substantially stabilizes training and reaches higher rewards than previous methods.
Abstract:RL post-training has become increasingly pivotal for improving diffusion policies, but existing diffusion policy-gradient methods are often unstable and cannot achieve reliable policy improvement. We identify the cause as the double-drift phenomenon: optimizing a variational surrogate can let the ELBO separate from the true log-likelihood, which then makes the resulting proxy policy gradient misaligned with the true policy gradient of expected return. We propose \textbf{DiPOD}, a diffusion policy optimization framework that maintains tight-bound behavior throughout training by interleaving self-distillation with policy-improving gradient updates. This leads to a simple and practical algorithm: augmenting each diffusion policy-gradient update with an on-policy ELBO regularizer. Across diffusion language model post-training and continuous-control diffusion policies, DiPOD substantially stabilizes training and reaches higher rewards than previous methods.
Machine Learning (cs.LG)
arXiv:2606.13795 [cs.LG]
(or arXiv:2606.13795v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.13795 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 213 words
Related coverage
iADD: Improving Alignment and Diversity in Diffusion Policy Optimization
arXiv · Oct 2, 2026
Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models
arXiv · Sep 2, 2026
Optimizers for Diffusion Models: A Controlled Benchmark
arXiv · Sep 22, 2026
Some remarks on the connection between generative diffusion and optimal transport
arXiv · Oct 12, 2026