aiqbench analysis
WOMBET addresses a critical bottleneck in robotic reinforcement learning by introducing a framework that jointly generates and utilizes prior data through world models. By filtering trajectories based on uncertainty penalties before online fine-tuning, the method significantly improves sample efficiency and stability across continuous control benchmarks without relying solely on fixed datasets. This approach offers a robust pathway for transferring skills from source tasks to target tasks while mitigating the high costs associated with real-world data collection in robotics.
WOMBET: World Model-Based Experience Transfer for Robust and Sample-efficient Reinforcement Learning
arXiv:2604.08958v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) in robotics is often limited by the cost and risk of data collection, motivating experience transfer from a source task to a target task. Offline-to-online RL leverages prior data but typically assumes a given fixed dataset and does not address how to generate reliable data for transfer. We propose World Model-Based Experience Transfer (WOMBET), a framework that jointly generates and utilizes prior data. WOMBET learns a world model in the source task and generates offline data via uncertainty-penalized planning, followed by filtering trajectories with high return and low epistemic uncertainty. It then performs online fine-tuning in the target task using adaptive sampling between offline and online data, enabling a stable transition from prior-driven initialization to task-specific adaptation. We show that the uncertainty-penalized objective provides a lower bound on the true return and derive a finite-sample error decomposition capturing distribution mismatch and approximation error. Empirically, WOMBET improves sample efficiency and final performance over strong baselines on continuous control benchmarks, demonstrating the benefit of jointly optimizing data generation and transfer.
Abstract:Reinforcement learning (RL) in robotics is often limited by the cost and risk of data collection, motivating experience transfer from a source task to a target task. Offline-to-online RL leverages prior data but typically assumes a given fixed dataset and does not address how to generate reliable data for transfer. We propose World Model-Based Experience Transfer (WOMBET), a framework that jointly generates and utilizes prior data. WOMBET learns a world model in the source task and generates offline data via uncertainty-penalized planning, followed by filtering trajectories with high return and low epistemic uncertainty. It then performs online fine-tuning in the target task using adaptive sampling between offline and online data, enabling a stable transition from prior-driven initialization to task-specific adaptation. We show that the uncertainty-penalized objective provides a lower bound on the true return and derive a finite-sample error decomposition capturing distribution mismatch and approximation error. Empirically, WOMBET improves sample efficiency and final performance over strong baselines on continuous control benchmarks, demonstrating the benefit of jointly optimizing data generation and transfer.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
arXiv:2604.08958 [cs.LG]
(or arXiv:2604.08958v3 [cs.LG] for this version)
Full text under the publisher's open licence · 187 words
Related coverage
The Machines that Make the Machines
NVIDIA · Oct 7, 2026
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · Oct 16, 2026
Self-Modeling Interventions Modulate Emergent Misalignment
LessWrong · Oct 5, 2026
ARROW: Augmented Replay for RObust World models
arXiv · Jun 12, 2026