aiqbench analysis
This research introduces UMM-RM, a novel architecture designed to mitigate reward hacking in reinforcement learning from human feedback by combining shared and routed experts into a single dense model. The approach effectively balances instruction-agnostic preference signals with fine-grained task modeling while eliminating the inference overhead typically associated with Mixture of Experts systems. By improving alignment stability under distribution shifts, this work addresses a critical vulnerability in current RLHF pipelines where policy models exploit spurious correlations in reward models.
UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking
arXiv:2512.00724v3 Announce Type: replace Abstract: Reward models (RMs) are critical to reinforcement learning from human feedback (RLHF). However, policy models can exploit biases or spurious correlations in dense RMs, causing reward hacking: RM scores rise while alignment with human preferences deteriorates, especially under distribution shift. We propose UMM-RM (Upcycle-and-Merge MoE Reward Model), which copies the feed-forward layers of an SFT checkpoint to initialize shared and routed experts, then trains the model on preference data. The always-active shared expert captures instruction-agnostic preference signals, while routed experts model fine-grained preferences across instructions or tasks. The experts are then merged into a single dense RM using learnable weights, preserving the robustness and exploitation resistance of expert diversity without MoE or ensemble inference overhead. Across multiple base models and preference datasets, UMM-RM improves preference accuracy, reduces reward hacking during PPO training, and achieves more stable preference alignment than standard dense RMs.
Abstract:Reward models (RMs) are critical to reinforcement learning from human feedback (RLHF). However, policy models can exploit biases or spurious correlations in dense RMs, causing reward hacking: RM scores rise while alignment with human preferences deteriorates, especially under distribution shift. We propose UMM-RM (Upcycle-and-Merge MoE Reward Model), which copies the feed-forward layers of an SFT checkpoint to initialize shared and routed experts, then trains the model on preference data. The always-active shared expert captures instruction-agnostic preference signals, while routed experts model fine-grained preferences across instructions or tasks. The experts are then merged into a single dense RM using learnable weights, preserving the robustness and exploitation resistance of expert diversity without MoE or ensemble inference overhead. Across multiple base models and preference datasets, UMM-RM improves preference accuracy, reduces reward hacking during PPO training, and achieves more stable preference alignment than standard dense RMs.
Machine Learning (cs.LG); Information Retrieval (cs.IR)
arXiv:2512.00724 [cs.LG]
(or arXiv:2512.00724v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2512.00724 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 237 words
Related coverage
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
Why do models task game?
AI Alignment Forum · Aug 6, 2026
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · Aug 3, 2026
Improving our alignment and security efforts
Anthropic · Aug 31, 2026