aiqbench analysis
This research addresses the critical issue of label noise in human preference data, which often obscures multi-dimensional aesthetic and semantic criteria into misleading binary signals. By introducing Semi-DPO, a semi-supervised method that iteratively refines pseudo-labels from clean consensus subsets, the authors demonstrate a robust way to align diffusion models with complex visual preferences without extra annotation costs. This advancement directly improves the reliability of Direct Preference Optimization in generative AI systems by mitigating gradient conflicts caused by imperfect training data.
Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization
arXiv:2604.24952v2 Announce Type: replace-cross Abstract: Human visual preferences are inherently multi-dimensional, encompassing aesthetics, detail fidelity, and semantic alignment. However, existing datasets provide only single, holistic annotations, resulting in severe label noise: images that excel in some dimensions but are deficient in others are simply marked as winner or loser. We theoretically demonstrate that compressing multi-dimensional preferences into binary labels generates conflicting gradient signals that misguide Diffusion Direct Preference Optimization (DPO). To address this, we propose Semi-DPO, a semi-supervised approach that treats consistent pairs as clean labeled data and conflicting ones as noisy unlabeled data. Our method starts by training on a consensus-filtered clean subset, then uses this model as an implicit classifier to generate pseudo-labels for the noisy set for iterative refinement. Experimental results demonstrate that Semi-DPO achieves state-of-the-art performance and significantly improves alignment with complex human preferences, without requiring additional human annotation or explicit reward models during training. We will release our code and models at: https://github.com/L-CodingSpace/semi-dpo
Abstract:Human visual preferences are inherently multi-dimensional, encompassing aesthetics, detail fidelity, and semantic alignment. However, existing datasets provide only single, holistic annotations, resulting in severe label noise: images that excel in some dimensions but are deficient in others are simply marked as winner or loser. We theoretically demonstrate that compressing multi-dimensional preferences into binary labels generates conflicting gradient signals that misguide Diffusion Direct Preference Optimization (DPO). To address this, we propose Semi-DPO, a semi-supervised approach that treats consistent pairs as clean labeled data and conflicting ones as noisy unlabeled data. Our method starts by training on a consensus-filtered clean subset, then uses this model as an implicit classifier to generate pseudo-labels for the noisy set for iterative refinement. Experimental results demonstrate that Semi-DPO achieves state-of-the-art performance and significantly improves alignment with complex human preferences, without requiring additional human annotation or explicit reward models during training. We will release our code and models at: this https URL
21 pages. Published as a conference paper at ICLR 2026
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2604.24952 [cs.CV]
(or arXiv:2604.24952v2 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2604.24952 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 263 words
Related coverage
JEPAMatch: Geometric Representation Shaping for Semi-Supervised Learning
arXiv · Aug 27, 2026
Mastering Agentic Techniques: AI Agent Customization
NVIDIA · May 20, 2026
Positive-Unlabeled Preference Optimization For Chest X-ray Report Generation
arXiv · Aug 7, 2026
Flick: Few Labels Text Classification using K-Aware Intermediate Learning in Multi-Task Low-Resource Languages
arXiv · Jul 28, 2026