aiqbench analysis
TechPulse take: This research addresses a core challenge in training diffusion language models by introducing RLDF, a method that improves policy loss estimation through optimized sampling of intermediate noisy states. The approach delivers consistent performance gains across multiple architectures and reasoning benchmarks, offering a scalable foundation for future reinforcement learning in this domain. A new open-source training framework, Drift, is released to support adoption of this technique.
Reinforcement Learning from Denoising Feedback
arXiv:2605.25638v2 Announce Type: replace-cross Abstract: Policy loss estimation remains a fundamental and long-standing challenge in reinforcement learning (RL) for diffusion language models (DLMs). We introduce Reinforcement Learning from Denoising Feedback (RLDF), a novel training paradigm that leverages feedback obtained from rollout and training processes to facilitate accurate and efficient policy loss estimation. To balance the trade-off between computational efficiency and estimation effectiveness, RLDF optimizes the model toward the clipped clean state from intermediate noisy states, combined with weighted timestep sampling over denoising timesteps. Extensive experiments demonstrate that RLDF achieves consistent and substantial improvements in both performance and generalizability across two representative DLM architectures, LLaDA and Dream, on multiple reasoning benchmarks. Our work lays a principled foundation for scalable reinforcement learning in diffusion language models. We build Drift, a training framework for DLMs, available at https://github.com/ant-research/Drift.
Abstract:Policy loss estimation remains a fundamental and long-standing challenge in reinforcement learning (RL) for diffusion language models (DLMs). We introduce Reinforcement Learning from Denoising Feedback (RLDF), a novel training paradigm that leverages feedback obtained from rollout and training processes to facilitate accurate and efficient policy loss estimation. To balance the trade-off between computational efficiency and estimation effectiveness, RLDF optimizes the model toward the clipped clean state from intermediate noisy states, combined with weighted timestep sampling over denoising timesteps. Extensive experiments demonstrate that RLDF achieves consistent and substantial improvements in both performance and generalizability across two representative DLM architectures, LLaDA and Dream, on multiple reasoning benchmarks. Our work lays a principled foundation for scalable reinforcement learning in diffusion language models. We build Drift, a training framework for DLMs, available at this https URL.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2605.25638 [cs.CL]
(or arXiv:2605.25638v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.25638 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 228 words
Related coverage
A Diffusion-Refined Planner with Reinforcement Learning Priors for Confined-Space Parking
arXiv · Aug 19, 2026
Where Success Breaks: Failure-Boundary Learning for Robust Vision-Language-Action Models
arXiv · Sep 9, 2026
Think Less, Act Early: Reinforced Latent Reasoning with Early Exit in Vision-Language-Action Models
arXiv · Jun 16, 2026
Conditional Generation of Creative Chess Puzzles with Diffusion Models
arXiv · Oct 1, 2026