analiza aiqbench
This research introduces U2A, a bi-level optimization framework that integrates machine unlearning into preference alignment for LLMs. By systematically quantifying the impact of removing negative examples rather than relying solely on costly positive datasets, it offers a more efficient alternative to standard reinforcement learning methods like RLHF. The approach demonstrates how targeted data removal can optimize model behavior while reducing computational overhead and dataset acquisition costs.
Leveraging Machine Unlearning for Cost-Efficient Preference Alignment
arXiv:2504.06659v2 Announce Type: replace Abstract: Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like reinforcement learning with human feedback face notable challenges. These approaches require high-quality datasets of positive preference examples, which are costly to obtain and computationally intensive. The LLM unlearning technique presents a promising alternative by directly removing the influence of negative examples. However, current research has primarily focused on empirical validation, lacking systematic quantitative analysis. To bridge this gap, we propose a framework linking PA with LLM unlearning. Through bi-level optimization, we first quantify how unlearning specific negative examples impacts PA performance. Our analysis reveals that these effects vary substantially across negative examples. Building on this insight, we pose a crucial question: how can we optimally select and weight negative examples for unlearning to maximize PA performance? To answer this, we propose Unlearning to Align (U2A), which leverages bi-level optimization to efficiently select and unlearn examples for optimal PA performance. We validate the proposed method through extensive experiments, with results confirming its effectiveness. Our code is available at https://github.com/muyiahhh/U2A.
Abstract:Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like reinforcement learning with human feedback face notable challenges. These approaches require high-quality datasets of positive preference examples, which are costly to obtain and computationally intensive. The LLM unlearning technique presents a promising alternative by directly removing the influence of negative examples. However, current research has primarily focused on empirical validation, lacking systematic quantitative analysis. To bridge this gap, we propose a framework linking PA with LLM unlearning. Through bi-level optimization, we first quantify how unlearning specific negative examples impacts PA performance. Our analysis reveals that these effects vary substantially across negative examples. Building on this insight, we pose a crucial question: how can we optimally select and weight negative examples for unlearning to maximize PA performance? To answer this, we propose Unlearning to Align (U2A), which leverages bi-level optimization to efficiently select and unlearn examples for optimal PA performance. We validate the proposed method through extensive experiments, with results confirming its effectiveness. Our code is available at this https URL.
Accepted by ICML 2026. 12 pages, 6 figures, and 4 tables. Code available at this https URL
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2504.06659 [cs.LG]
(or arXiv:2504.06659v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2504.06659 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 290 cuvinte
Subiecte conexe
Amazon Research Awards recipients announced
Amazon Science · 27 mai 2026
34 Amazon Research Awards Build on Trainium recipients announced
Amazon Science · 5 aug. 2026
Amazon is investing in the Lean Focused Research Organization
Amazon Science · 26 iul. 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · 2 oct. 2026