aiqbench analysis
This research addresses a critical flaw in current alignment methods by relaxing the assumption of a known link function between preferences and latent rewards, thereby mitigating bias caused by model misspecification. By formulating the problem as a semiparametric single-index model, the authors provide link-agnostic convergence guarantees and direct policy learning methods that remain robust even when structural parameters are unidentifiable. The work offers a theoretically grounded, practical advancement for improving the reliability of preference-based optimization in language models.
Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Model
arXiv:2512.21917v3 Announce Type: replace Abstract: Policy alignment to preference data typically assumes a known link function between observed preferences and latent rewards (e.g., Bradley-Terry model / logistic link). Misspecification of this link can bias inferred rewards and misalign learned policies. We study policy alignment under an unknown and unrestricted link function. We formulate an $f$-divergence-constrained reward maximization problem and show that realizability in a policy class induces a semiparametric single-index binary choice model, where a scalar policy-induced index captures all dependence on demonstrations and the remaining preference distribution is unrestricted. Rather than impose identifiability of structural parameters of such a model and estimate them, as in econometrics, we develop methods that directly learn policies, with the reward function implicit, analyzing error to the optimal policy and allowing for unidentifiable and nonparametric indices. We prove link-agnostic convergence guarantees in terms of generic function complexity measures and validate the methods and theory empirically. Code is available at https://github.com/causalml/spo/.
Abstract:Policy alignment to preference data typically assumes a known link function between observed preferences and latent rewards (e.g., Bradley-Terry model / logistic link). Misspecification of this link can bias inferred rewards and misalign learned policies. We study policy alignment under an unknown and unrestricted link function. We formulate an $f$-divergence-constrained reward maximization problem and show that realizability in a policy class induces a semiparametric single-index binary choice model, where a scalar policy-induced index captures all dependence on demonstrations and the remaining preference distribution is unrestricted. Rather than impose identifiability of structural parameters of such a model and estimate them, as in econometrics, we develop methods that directly learn policies, with the reward function implicit, analyzing error to the optimal policy and allowing for unidentifiable and nonparametric indices. We prove link-agnostic convergence guarantees in terms of generic function complexity measures and validate the methods and theory empirically. Code is available at this https URL.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Econometrics (econ.EM); Machine Learning (stat.ML)
arXiv:2512.21917 [cs.LG]
(or arXiv:2512.21917v3 [cs.LG] for this version)
Full text under the publisher's open licence · 172 words
Related coverage
Semiparametric Double Reinforcement Learning with Applications to Long-Term Causal Inference
arXiv · Aug 25, 2026
Minimax-Optimal Semiparametric Contextual Dynamic Pricing with Multimodal Revenue
arXiv · Aug 5, 2026
FLEXI-Haz: Semiparametric Deep Survival Inference Beyond Proportional Hazards
arXiv · Oct 21, 2026
Computationally efficient goodness-of-fit tests through kernelized Stein discrepancy
arXiv · Oct 7, 2026