analiza aiqbench
This paper improves credit assignment in reinforcement learning for LLM agents by reusing continuation evidence already present in a rollout batch, with no extra rollouts. The reported gains over GiGPO are modest (3.39 and 5.86 points on ALFWorld and WebShop) and come from a single 1.5B-parameter model, so broader validation across models and tasks is needed. For readers training agents, the practical point is that value estimates can be refined from existing batch data at no added sampling cost.
Action-Value Calibration via Continuation Sharing for Long-Horizon Agentic Reinforcement Learning
arXiv:2610.37195v1 Announce Type: cross Abstract: Reinforcement learning for long-horizon agents relies on feedback from sequences of interactions, making credit assignment to individual actions difficult. Existing methods such as GiGPO compare actions at shared states to compute relative advantages, but estimate each action's value from a single sampled continuation. The resulting local credit remains sensitive to decisions made later in the trajectory. To address this problem, we propose ConCal (Continuation Calibration), a recursive continuation-value estimator that incorporates continuation evidence already present in a rollout batch. ConCal matches visits from the same task by their incoming state-action-state transitions, combines shared continuation evidence with each visit's own transition estimate, and propagates calibrated values backward to refine earlier action-value estimates. This reduces dependence on individual sampled continuations and mitigates local credit distortions without additional rollouts. Held-out return diagnostics show lower error in within-group value differences and higher agreement with reference action orderings than estimates based on individual continuations. With Qwen2.5-1.5B-Instruct, ConCal improves success rates over GiGPO by 3.39 and 5.86 percentage points on ALFWorld and WebShop, respectively. Our code is available at https://github.com/Yingjian-Chen/ConCal.
Abstract:Reinforcement learning for long-horizon agents relies on feedback from sequences of interactions, making credit assignment to individual actions difficult. Existing methods such as GiGPO compare actions at shared states to compute relative advantages, but estimate each action's value from a single sampled continuation. The resulting local credit remains sensitive to decisions made later in the trajectory. To address this problem, we propose ConCal (Continuation Calibration), a recursive continuation-value estimator that incorporates continuation evidence already present in a rollout batch. ConCal matches visits from the same task by their incoming state-action-state transitions, combines shared continuation evidence with each visit's own transition estimate, and propagates calibrated values backward to refine earlier action-value estimates. This reduces dependence on individual sampled continuations and mitigates local credit distortions without additional rollouts. Held-out return diagnostics show lower error in within-group value differences and higher agreement with reference action orderings than estimates based on individual continuations. With Qwen2.5-1.5B-Instruct, ConCal improves success rates over GiGPO by 3.39 and 5.86 percentage points on ALFWorld and WebShop, respectively. Our code is available at this https URL.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.37195 [cs.LG]
(or arXiv:2610.37195v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.37195 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 273 cuvinte
Subiecte conexe
CIDER: Continual Interactive Distillation for Embodied Reinforcement Learning
arXiv · 25 aug. 2026
Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
arXiv · 10 iun. 2026
NeuroSynth: A Biologically Inspired Continual Reinforcement Learning Architecture for Mitigating Catastrophic Forgetting
arXiv · 3 aug. 2026
Continual Graph Multi-Agent Reinforcement Learning
arXiv · 8 oct. 2026