aiqbench analysis
Freeform Preference Learning addresses a critical bottleneck in robotic manipulation by enabling policies to optimize along multiple human-defined axes, such as speed or safety, rather than relying on sparse binary success signals. This approach leverages language-conditioned reward models to generate dense progress feedback and allows for dynamic behavior steering at test time without retraining the underlying policy. The method demonstrates significant performance gains in long-horizon tasks while introducing a new paradigm for composing complex behaviors from natural-language specifications.
Freeform Preference Learning for Robotic Manipulation
arXiv:2606.32027v3 Announce Type: replace-cross Abstract: Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at https://freeform-pl.github.io/fpl.website/
Abstract:Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at this https URL
Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2606.32027 [cs.RO]
(or arXiv:2606.32027v3 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2606.32027 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 286 words
Related coverage
LIBERO-PeRM: Benchmarking Personalized Robotic Manipulation
arXiv · Oct 17, 2026
Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation
arXiv · Jun 3, 2026
SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation
arXiv · Oct 16, 2026
Deconfusing Corrigibility
AI Alignment Forum · Oct 17, 2026