analiza aiqbench
This paper addresses a critical challenge in multi-agent reinforcement learning by proposing a new credit assignment framework that distinguishes between agents driving system performance and those merely reacting to rewards. By deriving gradient-corrected training signals from filtered feedback, the authors demonstrate improved accuracy on complex reasoning tasks like GSM8K without increasing generation costs. This work advances the theoretical foundations of training collaborative LLM systems, offering a practical solution to the inherent ambiguity in shared-reward environments.
Marginal-Contribution Policy Gradients under Filtered Feedback for Multi-Agent LLMs
arXiv:2604.22785v2 Announce Type: replace Abstract: We develop a unified treatment of credit assignment for RL training in multi-agent LLM systems. We show that observed reward alone cannot distinguish an agent that determines it from one that never affects it, and that standard shared-reward training performs exact gradient ascent on each agent's private utility rather than system performance. Moreover, we prove no single scalar per agent can consistently account for joint performance once agents interact. We thus develop the unique background-dependent notion of marginal contribution satisfying natural consistency requirements. From it we derive gradient-correct marginal contribution training signals, identify them from filtered feedback, and optimally allocate a budget of exact counterfactual evaluations against learned-signal error. Instantiated in GRPO, our signal improves routed GSM8K accuracy over winner-take-all training at no extra generation cost.
Abstract:We develop a unified treatment of credit assignment for RL training in multi-agent LLM systems. We show that observed reward alone cannot distinguish an agent that determines it from one that never affects it, and that standard shared-reward training performs exact gradient ascent on each agent's private utility rather than system performance. Moreover, we prove no single scalar per agent can consistently account for joint performance once agents interact. We thus develop the unique background-dependent notion of marginal contribution satisfying natural consistency requirements. From it we derive gradient-correct marginal contribution training signals, identify them from filtered feedback, and optimally allocate a budget of exact counterfactual evaluations against learned-signal error. Instantiated in GRPO, our signal improves routed GSM8K accuracy over winner-take-all training at no extra generation cost.
22 pages, 1 figure. Substantially revised version
Machine Learning (cs.LG)
arXiv:2604.22785 [cs.LG]
(or arXiv:2604.22785v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2604.22785 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 225 cuvinte
Subiecte conexe
Links #5: 2026/07
LessWrong · 31 iul. 2026
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · 5 oct. 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
Continual learning might make your blocking monitors nearly useless
AI Alignment Forum · 24 sept. 2026