Curriculum Learning for Safety Alignment

arXiv:2605.26315v2 Announce Type: replace Abstract: Direct Preference Optimization (DPO) is a widely used approach for safety alignment that aims to reduce harmful behaviors in large language models. However, prior work shows that it can be brittle and exhibits poor out-of-distribution (OOD) generalization. In this paper we investigate whether Curriculum Learning can improve the robustness of DPO-based safety alignment. We propose Staged-Competence, a curriculum-based framework that organizes preference data by difficulty, employs competence-based sampling, and progressively updates the reference model during training. Averaged across three model families and three training seeds, Staged-Competence reduces OOD harmful response rates by 13.6 percentage points and attack success rates across five jailbreak attacks by 16.5 percentage points, while preserving general capabilities and maintaining near-zero over-refusal. We further show that Staged-Competence: (1) matches baseline safety performance with only 75% of the training data, demonstrating improved data efficiency and (2) yields better separation between safe and unsafe responses. Staged-Competence is agnostic to the underlying policy optimization loss and can extend to other DPO variants and alignment domains beyond safety. Additionally, we find multiple inconsistencies in standard RLHF datasets and create a cleaner combined version, named PiKacHHU. Our code is available at https://github.com/Sandeep5500/curriculum-learning-for-safety and the dataset at https://huggingface.co/datasets/etrigan5500/PiKacHHU.

Abstract:Direct Preference Optimization (DPO) is a widely used approach for safety alignment that aims to reduce harmful behaviors in large language models. However, prior work shows that it can be brittle and exhibits poor out-of-distribution (OOD) generalization. In this paper we investigate whether Curriculum Learning can improve the robustness of DPO-based safety alignment. We propose Staged-Competence, a curriculum-based framework that organizes preference data by difficulty, employs competence-based sampling, and progressively updates the reference model during training. Averaged across three model families and three training seeds, Staged-Competence reduces OOD harmful response rates by 13.6 percentage points and attack success rates across five jailbreak attacks by 16.5 percentage points, while preserving general capabilities and maintaining near-zero over-refusal. We further show that Staged-Competence: (1) matches baseline safety performance with only 75% of the training data, demonstrating improved data efficiency and (2) yields better separation between safe and unsafe responses. Staged-Competence is agnostic to the underlying policy optimization loss and can extend to other DPO variants and alignment domains beyond safety. Additionally, we find multiple inconsistencies in standard RLHF datasets and create a cleaner combined version, named PiKacHHU. Our code is available at this https URL and the dataset at this https URL.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2605.26315 [cs.LG]

(or arXiv:2605.26315v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.26315 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Curriculum Learning for Safety Alignment

Related coverage