aiqbench analysis
Stochastic MeanFlow Policies (SMFP) address a critical bottleneck in reinforcement learning by enabling one-step generative control with explicit entropy regulation, overcoming the limitations of both Gaussian and iterative sampling baselines. This approach facilitates efficient multimodal action modeling and stable exploration in sparse-reward environments without sacrificing inference speed. The method represents a significant theoretical and practical advance for online off-policy algorithms requiring tractable entropy estimates.
Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent
arXiv:2605.21282v3 Announce Type: replace Abstract: Combining entropy regularisation with mirror descent in online off-policy reinforcement learning requires expressive policies with efficient sampling and explicit entropy control. Gaussian policies struggle to represent multimodal targets, whereas generative policies often require iterative sampling or lack tractable entropy estimates. We introduce Stochastic MeanFlow Policies (SMFP), a one-step generative policy class that combines MeanFlow-based noise-to-action mappings with Gaussian reparameterisation. The latent mapping captures diverse action modes, while a learnable conditional Gaussian scale provides an analytic entropy lower bound. We use this bound to regulate stochasticity alongside direct value guidance and advantage-weighted MeanFlow regression, enabling entropy-regularised mirror-descent learning with one-step policies. Across seven MuJoCo benchmarks, SMFP achieves strong performance against Gaussian and generative baselines with single-step inference. Additional experiments demonstrate multimodal action modelling and effective exploration in sparse-reward navigation.
Abstract:Combining entropy regularisation with mirror descent in online off-policy reinforcement learning requires expressive policies with efficient sampling and explicit entropy control. Gaussian policies struggle to represent multimodal targets, whereas generative policies often require iterative sampling or lack tractable entropy estimates. We introduce Stochastic MeanFlow Policies (SMFP), a one-step generative policy class that combines MeanFlow-based noise-to-action mappings with Gaussian reparameterisation. The latent mapping captures diverse action modes, while a learnable conditional Gaussian scale provides an analytic entropy lower bound. We use this bound to regulate stochasticity alongside direct value guidance and advantage-weighted MeanFlow regression, enabling entropy-regularised mirror-descent learning with one-step policies. Across seven MuJoCo benchmarks, SMFP achieves strong performance against Gaussian and generative baselines with single-step inference. Additional experiments demonstrate multimodal action modelling and effective exploration in sparse-reward navigation.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.21282 [cs.LG]
(or arXiv:2605.21282v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.21282 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 223 words
Related coverage
Flow-Map GRPO: Reinforcement Learning for Few-Step Flow-Map Generators via Anchored Stochastic Composition
arXiv · Sep 30, 2026
HybridFlow: A 2-NFE Generative Policy for Real-Time Robotic Manipulation
arXiv · Sep 22, 2026
NAVIRA: Decoupled Stochastic Remasking for Masked Diffusion Language Models
arXiv · Jun 5, 2026
Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control
arXiv · Sep 2, 2026