aiqbench analysis
SoftGAC addresses critical bottlenecks in maximum-entropy reinforcement learning by introducing a single-stochastic-network architecture with Gaussian bridge layers. This approach eliminates the need for iterative sampling and backpropagation through time, significantly reducing inference costs while maintaining competitive performance on continuous-control benchmarks. The method offers a more efficient compute-return tradeoff compared to existing diffusion and flow-based policies without sacrificing the ability to model complex action distributions.
Generative Actor-Critic with Soft Bridge Policies
arXiv:2605.08733v2 Announce Type: replace Abstract: Expressive generative policies such as diffusion and flow models are appealing for maximum-entropy (MaxEnt) online reinforcement learning because of their ability to model multimodal and highly non-Gaussian action distributions. However, training effective soft generative policies faces two obstacles that often arise together. First, marginal action densities are often unavailable, so existing methods typically rely on entropy bounds, heuristic proxies or approximations. Second, iterative shared-parameter samplers raise inference cost and require backpropagation through time over repeated network evaluations, increasing memory cost and potentially destabilizing policy optimization. These obstacles motivate us to seek a generative actor-critic method that exposes a tractable soft objective while generating each action in just one network evaluation. To this end, we propose soft generative actor-critic (SoftGAC), whose actor is a single stochastic feedforward network with internal Gaussian bridge layers. This explicit bridge structure allows us to lift the soft objective to path space, yielding an analytical path-KL regularizer against a high-entropy reference process for both actor improvement and the soft Bellman bootstrap, without estimating the endpoint entropy. Crucially, the actor generates each action in only one network evaluation under a total parameter budget comparable to strong baselines, substantially reducing inference cost relative to the existing iterative diffusion and flow policies. It also avoids long unrolled backpropagation through time. Across challenging continuous-control benchmarks, our method achieves competitive or superior returns compared with strong diffusion and flow policies, demonstrating a clear advantage in the compute-return tradeoff.
Abstract:Expressive generative policies such as diffusion and flow models are appealing for maximum-entropy (MaxEnt) online reinforcement learning because of their ability to model multimodal and highly non-Gaussian action distributions. However, training effective soft generative policies faces two obstacles that often arise together. First, marginal action densities are often unavailable, so existing methods typically rely on entropy bounds, heuristic proxies or approximations. Second, iterative shared-parameter samplers raise inference cost and require backpropagation through time over repeated network evaluations, increasing memory cost and potentially destabilizing policy optimization. These obstacles motivate us to seek a generative actor-critic method that exposes a tractable soft objective while generating each action in just one network evaluation. To this end, we propose soft generative actor-critic (SoftGAC), whose actor is a single stochastic feedforward network with internal Gaussian bridge layers. This explicit bridge structure allows us to lift the soft objective to path space, yielding an analytical path-KL regularizer against a high-entropy reference process for both actor improvement and the soft Bellman bootstrap, without estimating the endpoint entropy. Crucially, the actor generates each action in only one network evaluation under a total parameter budget comparable to strong baselines, substantially reducing inference cost relative to the existing iterative diffusion and flow policies. It also avoids long unrolled backpropagation through time. Across challenging continuous-control benchmarks, our method achieves competitive or superior returns compared with strong diffusion and flow policies, demonstrating a clear advantage in the compute-return tradeoff.
Machine Learning (cs.LG); Information Theory (cs.IT)
arXiv:2605.08733 [cs.LG]
(or arXiv:2605.08733v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.08733 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 332 words
Related coverage
Assessing Information Ecosystems : How Governments Can Get Ahead of Hybrid Threats
carnegieendowment.org · Aug 5, 2026
Constrained Flow Policy Updates: A Generalized Schr\"odinger Bridge View
arXiv · Sep 30, 2026
A Geometric Approach to Soft Actor-Critic with Zonotopes for Locomotion Learning
arXiv · Oct 10, 2026
Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data
arXiv · Sep 9, 2026