analiza aiqbench
This paper presents a flow-matching method for distributional reinforcement learning that couples current and successor return flows through shared base noise, aiming to fix boundary mismatch and reduce variance in bootstrapped targets. The key trade-off is a tunable parameter λ: λ=0 gives an unbiased target, while larger values accept controlled bias for lower variance. Results are reported on toy MRPs, OGBench, and D4RL, so practitioners should weigh the claimed stability and fidelity gains against the modest, benchmark-scale evidence.
Path-Coupled Bellman Flows for Distributional Reinforcement Learning
arXiv:2605.08253v3 Announce Type: replace Abstract: Distributional reinforcement learning (DRL) models the full return distribution, but existing finite-support or quantile-based methods rely on projections, while recent flow-based approaches can suffer from \emph{boundary mismatch} at the flow source or from \emph{high-variance} bootstrapping when current and successor noises are independent. We propose Path-Coupled Bellman Flows (PCBF), a continuous-time DRL method that learns return distributions with flow matching using \textbf{source-consistent Bellman-coupled paths}: the current path starts from the required base prior at $t{=}0$, reaches the Bellman target at $t{=}1$, and maintains a pathwise affine relation to the successor flow at intermediate times (without requiring time-$t$ marginals to satisfy a distributional Bellman fixed point for all $t$). PCBF couples current and successor return flows through shared base noise and uses a $\lambda$-parameterized control-variate target: $\lambda{=}0$ recovers an unbiased sample Bellman target, while $\lambda{>}0$ trades controlled bias for variance reduction. Experiments on analytically tractable MRPs, OGBench, and D4RL show improved distributional fidelity and training stability, and competitive offline RL performance.
Abstract:Distributional reinforcement learning (DRL) models the full return distribution, but existing finite-support or quantile-based methods rely on projections, while recent flow-based approaches can suffer from \emph{boundary mismatch} at the flow source or from \emph{high-variance} bootstrapping when current and successor noises are independent. We propose Path-Coupled Bellman Flows (PCBF), a continuous-time DRL method that learns return distributions with flow matching using \textbf{source-consistent Bellman-coupled paths}: the current path starts from the required base prior at $t{=}0$, reaches the Bellman target at $t{=}1$, and maintains a pathwise affine relation to the successor flow at intermediate times (without requiring time-$t$ marginals to satisfy a distributional Bellman fixed point for all $t$). PCBF couples current and successor return flows through shared base noise and uses a $\lambda$-parameterized control-variate target: $\lambda{=}0$ recovers an unbiased sample Bellman target, while $\lambda{>}0$ trades controlled bias for variance reduction. Experiments on analytically tractable MRPs, OGBench, and D4RL show improved distributional fidelity and training stability, and competitive offline RL performance.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.08253 [cs.LG]
(or arXiv:2605.08253v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.08253 Focus to learn more arXiv-issued DOI via DataCite
Proceedings of the 43rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 269 cuvinte
Subiecte conexe
Path-Coupled Bellman Flows for Distributional Reinforcement Learning
arXiv · 6 iun. 2026
A Diffusion-Refined Planner with Reinforcement Learning Priors for Confined-Space Parking
arXiv · 19 aug. 2026
Risk and Anomaly Identification for Distribution Network Optimal Operation Based on Reinforcement Learning and Uncertainty Quantification
arXiv · 4 sept. 2026
Online Inference for Quantile Temporal Difference Learning in Distributional Reinforcement Learning
arXiv · 17 aug. 2026