aiqbench analysis
Dual-Force addresses critical instability issues in offline reinforcement learning by replacing skill discriminators with successor features and Van der Waals force objectives, offering a more robust path to diverse policy generation without additional environment interaction. By leveraging pre-trained Functional Reward Encoding for zero-shot skill recall, this method effectively mitigates distribution shifts while maintaining high performance on locomotion and navigation benchmarks. This advancement represents a significant step forward in making offline imitation learning practical for real-world robotic applications where data collection is constrained.
Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraints
arXiv:2501.04426v2 Announce Type: replace-cross Abstract: Offline diversity maximization under imitation constraints can transform demonstration data into a set of distinct behavioral policies, improving robustness to distribution shift without additional environment interaction. In practice, however, existing offline approaches often rely on mutual-information objectives that require training a skill discriminator and can become unstable under the non-stationary rewards induced by alternating Lagrangian optimization. We introduce Dual-Force, an offline algorithm that (i) maximizes diversity using an off-policy estimator of a Van der Waals (VdW) force objective computed from successor features, eliminating the skill discriminator, and (ii) stabilizes training under non-stationary intrinsic rewards by conditioning the value function and policy on a pre-trained Functional Reward Encoding (FRE). The FRE code also enables zero-shot recall of every encountered skill via its associated latent representation, removing the need to pre-specify a fixed number of skills. On two Solo12 simulation benchmarks (locomotion and obstacle navigation), Dual-Force recovers diverse high-performing behaviors while matching a target expert state occupancy and improves robustness in adversarial obstacle variations.
Abstract:Offline diversity maximization under imitation constraints can transform demonstration data into a set of distinct behavioral policies, improving robustness to distribution shift without additional environment interaction. In practice, however, existing offline approaches often rely on mutual-information objectives that require training a skill discriminator and can become unstable under the non-stationary rewards induced by alternating Lagrangian optimization. We introduce Dual-Force, an offline algorithm that (i) maximizes diversity using an off-policy estimator of a Van der Waals (VdW) force objective computed from successor features, eliminating the skill discriminator, and (ii) stabilizes training under non-stationary intrinsic rewards by conditioning the value function and policy on a pre-trained Functional Reward Encoding (FRE). The FRE code also enables zero-shot recall of every encountered skill via its associated latent representation, removing the need to pre-specify a fixed number of skills. On two Solo12 simulation benchmarks (locomotion and obstacle navigation), Dual-Force recovers diverse high-performing behaviors while matching a target expert state occupancy and improves robustness in adversarial obstacle variations.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
arXiv:2501.04426 [cs.LG]
(or arXiv:2501.04426v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2501.04426 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 259 words
Related coverage
Q-Regularized Generative Auto-Bidding: From Suboptimal Trajectories to Optimal Policies
arXiv · Aug 19, 2026
Against Export Controls (and China Threat Models)
LessWrong · Sep 24, 2026
Dharma vs Adharma - The Perplexity of Being Alive
hillpost.in · Aug 25, 2026
News
Quantum Computing Report · Jun 7, 2026