analiza aiqbench
This paper proposes using LLMs to decompose long, sparse-reward tasks into subtasks and to derive dense rewards from a progress function via potential-based reward shaping, which the authors say preserves optimality and convergence. The claimed gains are demonstrated on robotics control benchmarks against LLM planning and reward-generation baselines. For practitioners, the key questions are how robust the progress function is outside those benchmarks and what the ongoing cost of LLM queries adds to training. The public code makes those checks feasible.
Progress Reward Model for Reinforcement Learning via Large Language Models
arXiv:2610.22056v1 Announce Type: new Abstract: Traditional reinforcement learning (RL) algorithms face significant limitations in handling long-term tasks with sparse rewards. Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-based methods require extensive human feedback or exhaustive searching due to the complexity of tasks. In this paper, we propose the Progress Reward Model for RL (PRM4RL), a novel framework that integrates task planning and dense reward to enhance RL. For high-level planning, a complex task is decomposed into a series of simple manageable subtasks, with a subtask-oriented, fine-grained progress function designed to monitor task execution progress. For low-level reward generation, inspired by potential-based reward shaping, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization. Experimental results on robotics control tasks demonstrate that our approach outperforms both LLM-based planning and reward methods, achieving state-of-the-art performance. Code is available at https://github.com/deng-ai-lab/PRM4RL.
Abstract:Traditional reinforcement learning (RL) algorithms face significant limitations in handling long-term tasks with sparse rewards. Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-based methods require extensive human feedback or exhaustive searching due to the complexity of tasks. In this paper, we propose the Progress Reward Model for RL (PRM4RL), a novel framework that integrates task planning and dense reward to enhance RL. For high-level planning, a complex task is decomposed into a series of simple manageable subtasks, with a subtask-oriented, fine-grained progress function designed to monitor task execution progress. For low-level reward generation, inspired by potential-based reward shaping, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization. Experimental results on robotics control tasks demonstrate that our approach outperforms both LLM-based planning and reward methods, achieving state-of-the-art performance. Code is available at this https URL.
20 pages, 10 figures. Published at NeurIPS 2025
Artificial Intelligence (cs.AI)
arXiv:2610.22056 [cs.AI]
(or arXiv:2610.22056v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.22056 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.52202/085713-3370 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 293 cuvinte
Subiecte conexe
What We're Up Against: An AI Safety Crash Course
LessWrong · 24 sept. 2026
The generative AI customization spectrum: From prompt engineering to custom models on AWS
AWS · 14 sept. 2026
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · 23 sept. 2026
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · 12 iun. 2026