aiqbench analysis
This arXiv paper studies active learning for collecting human preference labels, a costly bottleneck in aligning LLMs. It argues that classical experimental-design criteria like G- and D-optimality are poorly matched to preference learning, and proposes an instance-dependent label-complexity guarantee plus a simpler greedy method. Its evaluation on real and synthetic preference data reports better sample efficiency, though the reported optimality holds only up to a constant tied to the logit bound.
Towards Optimality in Active Preference Learning
arXiv:2602.01581v2 Announce Type: replace Abstract: Aligning large language models (LLMs) depends on high-quality datasets of human preference labels, which are costly to collect. Although active learning has been studied to improve sample efficiency relative to passive collection, many existing approaches adopt classical experimental design criteria such as G- or D-optimality. These objectives are not tailored to the structure of preference learning, leaving open the design of problem-specific algorithms. In this work, we identify a simple intuition specific to preference learning that calls into question the suitability of these existing design objectives. Motivated by this insight, we propose two active learning algorithms. The first provides the first instance-dependent label complexity guarantee for this setting, and the second is a simple, practical greedy method. We show the optimality of our instance-dependent term up to a constant depending on the logit bound, compared with a lower-bound characteristic. We evaluate our algorithms on real-world preference datasets and synthetic data observing improved sample efficiency compared to existing methods.
Abstract:Aligning large language models (LLMs) depends on high-quality datasets of human preference labels, which are costly to collect. Although active learning has been studied to improve sample efficiency relative to passive collection, many existing approaches adopt classical experimental design criteria such as G- or D-optimality. These objectives are not tailored to the structure of preference learning, leaving open the design of problem-specific algorithms. In this work, we identify a simple intuition specific to preference learning that calls into question the suitability of these existing design objectives. Motivated by this insight, we propose two active learning algorithms. The first provides the first instance-dependent label complexity guarantee for this setting, and the second is a simple, practical greedy method. We show the optimality of our instance-dependent term up to a constant depending on the logit bound, compared with a lower-bound characteristic. We evaluate our algorithms on real-world preference datasets and synthetic data observing improved sample efficiency compared to existing methods.
Machine Learning (cs.LG)
arXiv:2602.01581 [cs.LG]
(or arXiv:2602.01581v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.01581 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 250 words
Related coverage
Deconfusing Corrigibility
AI Alignment Forum · Oct 17, 2026
Measuring Reward-Seeking via Contrastive Belief Updates
LessWrong · Jul 21, 2026
The Most Forbidden Technique is not always forbidden
LessWrong · Jul 18, 2026
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · Sep 23, 2026