aiqbench analysis
This paper introduces Group-Marginalized Advantage Estimation (GMAE), a method designed to improve self-rewarding reinforcement learning by aggregating reward signals across multiple group contexts rather than relying on a single realization. By addressing the instability caused by random sampling in ensemble-based methods, GMAE enables large language models to achieve more reliable policy optimization and cross-domain generalization without human labels. The approach demonstrates stable learning with low additional computational costs across various RL backbones and datasets.
Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolving
arXiv:2609.36750v1 Announce Type: cross Abstract: Self-rewarding reinforcement learning (RL) enables large language models (LLMs) to self-evolve without human labels. Existing ensemble-based methods construct reward references from rollout groups and assign rewards accordingly. However, a response's reward representation also depends on its randomly sampled group context, i.e., the other responses in its group. Using only one group-context realization may miss desired reward signals and provide unreliable guidance for policy optimization. To address this issue, we propose Group-Marginalized Advantage Estimation (GMAE), which aggregates reward realizations across possible contexts into a response-level distribution and estimates expected advantages. Experiments across eight benchmarks and four base models demonstrate strong performance and cross-domain generalization. GMAE also exhibits stable learning, low extra cost, and good applicability across training datasets and RL backbones.
Abstract:Self-rewarding reinforcement learning (RL) enables large language models (LLMs) to self-evolve without human labels. Existing ensemble-based methods construct reward references from rollout groups and assign rewards accordingly. However, a response's reward representation also depends on its randomly sampled group context, i.e., the other responses in its group. Using only one group-context realization may miss desired reward signals and provide unreliable guidance for policy optimization. To address this issue, we propose Group-Marginalized Advantage Estimation (GMAE), which aggregates reward realizations across possible contexts into a response-level distribution and estimates expected advantages. Experiments across eight benchmarks and four base models demonstrate strong performance and cross-domain generalization. GMAE also exhibits stable learning, low extra cost, and good applicability across training datasets and RL backbones.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2609.36750 [cs.LG]
(or arXiv:2609.36750v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.36750 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 221 words
Related coverage
World Models and Yann LeCun’s Bet Against Large Language Models
Quantum Zeitgeist · Jul 17, 2026
Does distilling Claude carry the persona with it?
LessWrong · Jul 24, 2026
Problems of Proliferation
LessWrong · Sep 29, 2026
Echoverse: Deep, evolving environments for computer-use agents
Microsoft Research · Jul 30, 2026