Disentangling Optimization Scale from Preference Scale in DPO

arXiv:2608.27032v1 Announce Type: new Abstract: Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient $\beta$ commonly interpreted as controlling the KL constraint to a reference policy. We show that $\beta$ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size. As a consequence, at a fixed learning rate the achieved policy deviation is non-monotone in $\beta$: it vanishes in a dead zone at small $\beta$, reaches a peak at an intermediate value, and decreases again for larger $\beta$. Moreover, standard DPO loss values are not comparable across $\beta$: runs with nearly identical loss curves can differ several-fold in KL divergence from the reference model. This entanglement obscures the role of $\beta$, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling. We propose a centered-softplus reformulation that is argmin-equivalent to DPO for $\beta>0$, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable. The normalized centered-softplus objective also admits a continuous $\beta\to0$ endpoint that reduces to a linear preference-margin objective.

Abstract:Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient $\beta$ commonly interpreted as controlling the KL constraint to a reference policy. We show that $\beta$ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size. As a consequence, at a fixed learning rate the achieved policy deviation is non-monotone in $\beta$: it vanishes in a dead zone at small $\beta$, reaches a peak at an intermediate value, and decreases again for larger $\beta$. Moreover, standard DPO loss values are not comparable across $\beta$: runs with nearly identical loss curves can differ several-fold in KL divergence from the reference model. This entanglement obscures the role of $\beta$, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling. We propose a centered-softplus reformulation that is argmin-equivalent to DPO for $\beta>0$, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable. The normalized centered-softplus objective also admits a continuous $\beta\to0$ endpoint that reduces to a linear preference-margin objective.

Machine Learning (cs.LG)

arXiv:2608.27032 [cs.LG]

(or arXiv:2608.27032v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2608.27032 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

Disentangling Optimization Scale from Preference Scale in DPO

Subiecte conexe