analiza aiqbench
This paper proposes a refinement to GRPO, a reinforcement learning method for fine-tuning language models. Instead of giving every token in a response the same advantage signal, DATC weights each token by how well its policy update aligns with the reward model's gradient. The reported gains are over GRPO in four settings across two model families, so readers should check the benchmarks and compute costs before judging how general the improvement is.
Direction-Aware Token Credit Assignment with Reward-Model Gradients
arXiv:2610.32772v1 Announce Type: new Abstract: GRPO shares a response-level advantage across tokens, leaving unresolved how strongly each token should contribute to learning. Updates induced by the shared advantage reinforce or suppress the sampled token at each position, but these updates do not always point toward increasing reward at the token level. This motivates a work-based view of credit: just as work depends on the component of force along a displacement, token credit should reflect the local reward change along the policy update direction. We introduce Direction-Aware Token Credit Assignment (DATC), which maps frozen reward-model gradients into policy logit space and allocates weights by evaluating the alignment between the reward direction and GRPO's signed sampled-action update direction, while preserving the sign and mean of the response-level advantage. DATC outperforms GRPO in all four settings across two model families.
Abstract:GRPO shares a response-level advantage across tokens, leaving unresolved how strongly each token should contribute to learning. Updates induced by the shared advantage reinforce or suppress the sampled token at each position, but these updates do not always point toward increasing reward at the token level. This motivates a work-based view of credit: just as work depends on the component of force along a displacement, token credit should reflect the local reward change along the policy update direction. We introduce Direction-Aware Token Credit Assignment (DATC), which maps frozen reward-model gradients into policy logit space and allocates weights by evaluating the alignment between the reward direction and GRPO's signed sampled-action update direction, while preserving the sign and mean of the response-level advantage. DATC outperforms GRPO in all four settings across two model families.
28 pages, 7 figures. Project page: this https URL
Machine Learning (cs.LG)
arXiv:2610.32772 [cs.LG]
(or arXiv:2610.32772v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.32772 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 235 cuvinte