analiza aiqbench
STITCH-OPE introduces a novel generative framework using denoising diffusion to solve off-policy evaluation challenges in high-dimensional, long-horizon domains like robotics and healthcare. By stitching partial trajectories and refining the guidance process, this method theoretically achieves an exponential reduction in variance compared to existing approaches. The technique offers significant practical value for safely evaluating policies where direct environmental interaction is too costly or dangerous.
STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation
arXiv:2505.20781v2 Announce Type: replace-cross Abstract: Off-policy evaluation (OPE) estimates the performance of a target policy using offline data collected from a behavior policy, and is crucial in domains such as robotics or healthcare where direct interaction with the environment is costly or unsafe. Existing OPE methods are ineffective for high-dimensional, long-horizon problems, due to exponential blow-ups in variance from importance weighting or compounding errors from learned dynamics models. To address these challenges, we propose STITCH-OPE, a model-based generative framework that leverages denoising diffusion for long-horizon OPE in high-dimensional state and action spaces. Starting with a diffusion model pre-trained on the behavior data, STITCH-OPE generates synthetic trajectories from the target policy by guiding the denoising process using the score function of the target policy. STITCH-OPE proposes two technical innovations that make it advantageous for OPE: (1) prevents over-regularization by subtracting the score of the behavior policy during guidance, and (2) generates long-horizon trajectories by stitching partial trajectories together end-to-end. We provide a theoretical guarantee that under mild assumptions, these modifications result in an exponential reduction in variance versus long-horizon trajectory diffusion. Experiments on the D4RL and OpenAI Gym benchmarks show substantial improvement in mean squared error, correlation, and regret metrics compared to state-of-the-art OPE methods.
Abstract:Off-policy evaluation (OPE) estimates the performance of a target policy using offline data collected from a behavior policy, and is crucial in domains such as robotics or healthcare where direct interaction with the environment is costly or unsafe. Existing OPE methods are ineffective for high-dimensional, long-horizon problems, due to exponential blow-ups in variance from importance weighting or compounding errors from learned dynamics models. To address these challenges, we propose STITCH-OPE, a model-based generative framework that leverages denoising diffusion for long-horizon OPE in high-dimensional state and action spaces. Starting with a diffusion model pre-trained on the behavior data, STITCH-OPE generates synthetic trajectories from the target policy by guiding the denoising process using the score function of the target policy. STITCH-OPE proposes two technical innovations that make it advantageous for OPE: (1) prevents over-regularization by subtracting the score of the behavior policy during guidance, and (2) generates long-horizon trajectories by stitching partial trajectories together end-to-end. We provide a theoretical guarantee that under mild assumptions, these modifications result in an exponential reduction in variance versus long-horizon trajectory diffusion. Experiments on the D4RL and OpenAI Gym benchmarks show substantial improvement in mean squared error, correlation, and regret metrics compared to state-of-the-art OPE methods.
Robotics (cs.RO); Machine Learning (cs.LG)
arXiv:2505.20781 [cs.RO]
(or arXiv:2505.20781v2 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2505.20781 Focus to learn more arXiv-issued DOI via DataCite
https://doi.org/10.52202/085713-0242 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 303 cuvinte
Subiecte conexe
Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents
arXiv · 21 iul. 2026
Develop Humanoid Robot Policies End-to-End with NVIDIA Isaac GR00T
NVIDIA · 7 iul. 2026
LeRobot v0.6.0: Imagine, Evaluate, Improve
Hugging Face · 7 iul. 2026
Latent Reward Registers for Diffusion Preference Alignment
arXiv · 5 aug. 2026