Demystifying the Unreasonable Effectiveness of Greedy Alignment Methods

arXiv:2604.17207v2 Announce Type: replace Abstract: Greedy AI alignment methods based on Bradley-Terry reward-model fitting and KL-regularized Alignment are remarkably effective in both offline and online preference-learning pipelines, yet existing O(log T) online and O(1/epsilon) offline KL-regularized regret rate guarantees can seem pessimistic relative to their empirical performance. We argue that this mismatch reflects a distinction between two learning targets: KL-regularized regret measures how fast we can recover the KL-regularized aligned policy, whereas we often only ask how fast we can learn the reward function that induces the correct best response. To isolate this reward-learning component, we study the traditional temperature-zero regret criterion, which evaluates only the top-ranked response at inference time. Under realizability, compactness, and a uniform best-response margin over the fitted reward class, we prove that exact reference-logged offline RLHF achieves exponentially small expected temperature-zero regret, and therefore reaches expected regret at most epsilon with O(log(1\epsilon)) offline comparisons. We also prove that exact greedy online RLHF achieves bounded O(1) cumulative temperature-zero regret; the same guarantee extends to its equivalent pairwise online DPO formulation. By separating selector identification from full soft-policy recovery, our results provide a sharper theoretical explanation for the empirical efficiency of greedy alignment.

Abstract:Greedy AI alignment methods based on Bradley-Terry reward-model fitting and KL-regularized Alignment are remarkably effective in both offline and online preference-learning pipelines, yet existing O(log T) online and O(1/epsilon) offline KL-regularized regret rate guarantees can seem pessimistic relative to their empirical performance. We argue that this mismatch reflects a distinction between two learning targets: KL-regularized regret measures how fast we can recover the KL-regularized aligned policy, whereas we often only ask how fast we can learn the reward function that induces the correct best response. To isolate this reward-learning component, we study the traditional temperature-zero regret criterion, which evaluates only the top-ranked response at inference time. Under realizability, compactness, and a uniform best-response margin over the fitted reward class, we prove that exact reference-logged offline RLHF achieves exponentially small expected temperature-zero regret, and therefore reaches expected regret at most epsilon with O(log(1\epsilon)) offline comparisons. We also prove that exact greedy online RLHF achieves bounded O(1) cumulative temperature-zero regret; the same guarantee extends to its equivalent pairwise online DPO formulation. By separating selector identification from full soft-policy recovery, our results provide a sharper theoretical explanation for the empirical efficiency of greedy alignment.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computational Complexity (cs.CC); Computation and Language (cs.CL)

arXiv:2604.17207 [cs.LG]

(or arXiv:2604.17207v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2604.17207 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Demystifying the Unreasonable Effectiveness of Greedy Alignment Methods

Related coverage