analiza aiqbench
QAMM addresses the latency bottleneck in flow-based policies by leveraging adjoint matching to learn finite-interval transport directly, enabling effective two-step action generation without iterative sampling. This approach combines critic-derived gradients with average-velocity learning to achieve competitive performance on offline reinforcement learning benchmarks while reducing network evaluations. The method demonstrates a significant efficiency gain for scenarios requiring rapid decision-making in complex environments like HumanoidMaze.
QAMM: Adjoint MeanFlow Matching for Few-Step Offline Reinforcement Learning
arXiv:2609.34497v1 Announce Type: new Abstract: Flow policies can model rich action distributions, but their iterative sampling limits decision speed. Adjoint matching uses the critic's action gradient to improve a flow policy without backpropagating through its sampling trajectory, yet its supervision is defined for instantaneous velocities. We propose QAMM, a method that turns the critic-derived adjoint signal into supervision for MeanFlow's average velocity. The resulting policy learns finite-interval transport directly and generates actions with few network evaluations. We derive the adjoint MeanFlow target, specify its gradient boundaries, and train it with an offline actor-critic. On ten HumanoidMaze tasks, QAMM produces effective two-call policies and achieves competitive performance against strong flow-policy baselines. These results show that adjoint-based Q optimization can be combined with average-velocity learning to obtain expressive offline policies with few-step action generation.
Abstract:Flow policies can model rich action distributions, but their iterative sampling limits decision speed. Adjoint matching uses the critic's action gradient to improve a flow policy without backpropagating through its sampling trajectory, yet its supervision is defined for instantaneous velocities. We propose QAMM, a method that turns the critic-derived adjoint signal into supervision for MeanFlow's average velocity. The resulting policy learns finite-interval transport directly and generates actions with few network evaluations. We derive the adjoint MeanFlow target, specify its gradient boundaries, and train it with an offline actor-critic. On ten HumanoidMaze tasks, QAMM produces effective two-call policies and achieves competitive performance against strong flow-policy baselines. These results show that adjoint-based Q optimization can be combined with average-velocity learning to obtain expressive offline policies with few-step action generation.
Machine Learning (cs.LG)
arXiv:2609.34497 [cs.LG]
(or arXiv:2609.34497v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.34497 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 221 cuvinte
Subiecte conexe
Trust Region Q Adjoint Matching
arXiv · 10 oct. 2026
Diffusion Policy Improvement with Proposal-Conditioned Refinement Flows
arXiv · 30 sept. 2026
Simple Actors and Deep Critics for Scalable Reinforcement Learning
arXiv · 28 aug. 2026
Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning
arXiv · 10 iun. 2026