aiqbench analysis
T2SPO addresses the sparse reward problem in agentic reinforcement learning by leveraging successful historical trajectories to generate step-level feedback signals. By using a pretrained TabPFN regressor to estimate remaining distance to success, this method provides auxiliary credit that guides policy optimization more effectively than outcome-only supervision. Experimental results on LLMs demonstrate improved task success rates over standard GRPO approaches, offering a practical advancement for training complex multi-step agents.
T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
arXiv:2610.00388v1 Announce Type: cross Abstract: Reinforcement learning enables large language model (LLM) agents to learn multi-step behaviors through interaction with their environments. However, rewards in many interactive tasks reflect only the final outcome, providing limited guidance on which intermediate decisions advance the task. Successful training trajectories contain intermediate states that can provide supervision for subsequent interactions. We introduce Trajectory-to-Step Policy Optimization (T2SPO), a method that uses past interaction trajectories to provide step-level feedback for policy learning. T2SPO derives remaining-distance targets from successful trajectories and pairs them with representations of the states visited along the way. Conditioned on these examples, a pretrained TabPFN regressor estimates the remaining distance to success at each state of a new rollout. Changes in this distance estimate across consecutive states yield auxiliary credit for agent steps alongside task-level supervision. As training proceeds, newly completed trajectories refresh the estimator's context, incorporating new experience without updating its parameters. Experiments with 1.5B and 7B language models on ALFWorld and WebShop show that T2SPO consistently improves overall task success over GRPO.
Abstract:Reinforcement learning enables large language model (LLM) agents to learn multi-step behaviors through interaction with their environments. However, rewards in many interactive tasks reflect only the final outcome, providing limited guidance on which intermediate decisions advance the task. Successful training trajectories contain intermediate states that can provide supervision for subsequent interactions. We introduce Trajectory-to-Step Policy Optimization (T2SPO), a method that uses past interaction trajectories to provide step-level feedback for policy learning. T2SPO derives remaining-distance targets from successful trajectories and pairs them with representations of the states visited along the way. Conditioned on these examples, a pretrained TabPFN regressor estimates the remaining distance to success at each state of a new rollout. Changes in this distance estimate across consecutive states yield auxiliary credit for agent steps alongside task-level supervision. As training proceeds, newly completed trajectories refresh the estimator's context, incorporating new experience without updating its parameters. Experiments with 1.5B and 7B language models on ALFWorld and WebShop show that T2SPO consistently improves overall task success over GRPO.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.00388 [cs.LG]
(or arXiv:2610.00388v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.00388 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 263 words
Related coverage
When Partial Credit Opposes Completion
Snorkel AI Blog · Oct 15, 2026
Mastering Agentic Techniques: AI Agent Reinforcement Learning
NVIDIA · Jul 1, 2026
Fine-tune a search agent with multi-turn RL on Amazon SageMaker AI
AWS · Oct 2, 2026
Granite 4.2 LLMs: How They're Built
Hugging Face · Aug 25, 2026