analiza aiqbench
This research addresses the scarcity of human preference data in reward modeling by introducing MARS, an adaptive framework that prioritizes low-margin pairs and refines semantic contrasts before synthesizing samples. The study demonstrates improved alignment performance across multiple datasets without relying solely on external judges or simple augmentation techniques.
MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling
arXiv:2602.17658v4 Announce Type: replace Abstract: Reward modeling is central to RLHF, RLAIF, and PPO-based alignment, but its reliability is often limited by scarce and heterogeneous human preference data. In this paper, we introduce MARS (Margin and Semantic-Aware Data Augmentation for Reward Modeling), an adaptive augmentation framework for controlled low-resource reward modeling. MARS allocates more augmentation to low-margin preference pairs and uses semantic-distance-based refinement to improve chosen-rejected contrast before generating synthetic preference samples. Across three preference datasets, two reward-model backbones, and downstream alignment evaluations, MARS improves average RewardBench performance and alignment win rates over uniform augmentation, WoN, and AdaBoost-style baselines. Ablations and independent-judge evaluations suggest that the gains are not solely explained by semantic refinement alone or GPT-4.1 judge coupling.
Abstract:Reward modeling is central to RLHF, RLAIF, and PPO-based alignment, but its reliability is often limited by scarce and heterogeneous human preference data. In this paper, we introduce MARS (Margin and Semantic-Aware Data Augmentation for Reward Modeling), an adaptive augmentation framework for controlled low-resource reward modeling. MARS allocates more augmentation to low-margin preference pairs and uses semantic-distance-based refinement to improve chosen-rejected contrast before generating synthetic preference samples. Across three preference datasets, two reward-model backbones, and downstream alignment evaluations, MARS improves average RewardBench performance and alignment win rates over uniform augmentation, WoN, and AdaBoost-style baselines. Ablations and independent-judge evaluations suggest that the gains are not solely explained by semantic refinement alone or GPT-4.1 judge coupling.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Theory (cs.IT)
arXiv:2602.17658 [cs.LG]
(or arXiv:2602.17658v4 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.17658 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 212 cuvinte
Subiecte conexe
CoBRA: Learning Tool-Use Boundaries via Counterfactual Margins
arXiv · 2 sept. 2026
Contemporary AI lacks the imagination to diverge or negate in science
arXiv · 10 iun. 2026
Contemporary AI lacks the imagination to diverge or negate in science
arXiv · 9 iun. 2026
Artmarket.com: H1 2026 Financial Report -- One-of-a - Kind AI Systems Intuitive Artmarket and Blind Spot Will Drive the Next Phase of Growth for
australiannews.net · 29 sept. 2026