aiqbench analysis
Bootstrapped Flow Q-Learning (BFQ) addresses the computational brittleness of diffusion-based offline reinforcement learning by enabling accurate single-step action generation without auxiliary networks or distillation. By leveraging a divide-and-conquer approach to estimate displacement vectors, the method significantly reduces training and inference costs while maintaining high performance on D4RL benchmarks. This advancement suggests that complex multi-step denoising may not be necessary for optimal results, offering a more efficient path forward for offline RL systems.
Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning
arXiv:2606.10613v1 Announce Type: new Abstract: Diffusion-based Q-learning has emerged as a powerful paradigm for offline reinforcement learning, but its reliance on multi-step denoising makes both training and inference computationally expensive and brittle. Recent efforts to accelerate diffusion Q-learning toward single-step action generation typically introduce auxiliary networks, policy distillation, or multi-phase training, which frequently compromise simplicity, stability, or performance. To address these limitations, we introduce Bootstrapped Flow Q-Learning (BFQ), a novel framework that enables accurate single-step action generation during both training and inference, without auxiliary networks or distillation procedures. BFQ adopts a divide-and-conquer view of the displacement vector along the flow path: it begins by learning short-range displacements that can be accurately estimated from the Flow Matching marginal velocity, and bootstraps these components to directly learn a noise-to-action mapping in a single step. This formulation eliminates multi-step denoising, resulting in a learning procedure that is substantially faster, simpler, and more robust. Extensive D4RL evaluations show that BFQ improves performance while significantly reducing computational cost compared to multi-step diffusion baselines, demonstrating that single-step action generation suffices for high-performance offline Reinforcement Learning.
Abstract:Diffusion-based Q-learning has emerged as a powerful paradigm for offline reinforcement learning, but its reliance on multi-step denoising makes both training and inference computationally expensive and brittle. Recent efforts to accelerate diffusion Q-learning toward single-step action generation typically introduce auxiliary networks, policy distillation, or multi-phase training, which frequently compromise simplicity, stability, or performance. To address these limitations, we introduce Bootstrapped Flow Q-Learning (BFQ), a novel framework that enables accurate single-step action generation during both training and inference, without auxiliary networks or distillation procedures. BFQ adopts a divide-and-conquer view of the displacement vector along the flow path: it begins by learning short-range displacements that can be accurately estimated from the Flow Matching marginal velocity, and bootstraps these components to directly learn a noise-to-action mapping in a single step. This formulation eliminates multi-step denoising, resulting in a learning procedure that is substantially faster, simpler, and more robust. Extensive D4RL evaluations show that BFQ improves performance while significantly reducing computational cost compared to multi-step diffusion baselines, demonstrating that single-step action generation suffices for high-performance offline Reinforcement Learning.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2606.10613 [cs.LG]
(or arXiv:2606.10613v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.10613 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 271 words
Related coverage
The Alignment Problem: Computer Science Meets Analytic Philosophy
LessWrong · Oct 18, 2026
NaP-Control: Navigating Diffusion Prior for Versatile and Fast Character Control
arXiv · Jul 21, 2026
Developing Healthcare Robotics with GPU-Native Medical Physics Simulation
NVIDIA · Jul 28, 2026
Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments
arXiv · Oct 7, 2026