Selective Successor Eligibility Traces for RLVR

Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) has become a key approach for improving the reasoning abilities of large language models. However, widely used critic-free algorithms such as Group Relative Policy Optimization (GRPO) suffer from coarse-grained temporal credit assignment, as they assign the same trajectory-level advantage to all tokens and fail to distinguish the contributions of different reasoning steps. To address this challenge, we propose Selective Successor Eligibility Traces (S-trace), an critic-free eligibility trace method that enables more effective credit assignment. S-trace introduces successor importance reweighting to recover downstream credit signals under PPO-style clipping and propagates them to selected historical tokens through entropy-based token selection. Theoretically, we establish an upper bound on its discrepancy from the raw eligibility traces-based policy gradient estimator, jointly controlled by the KL divergence and the trace decay factor $\lambda$. Experiments show that S-trace achieves absolute gains in average pass@16 of 1.80\% and 6.17\% over GRPO, and 2.32\% and 0.87\% over trace-style GRPO($\lambda$), on Qwen3-1.7B and Qwen3-4B, respectively, while maintaining robust gains on Qwen3-8B and producing shorter responses.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2605.05965 [cs.LG]

(or arXiv:2605.05965v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.05965 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.