analiza aiqbench
This research establishes a theoretical framework where LLM-derived reward signals function as bounded potential functions within reinforcement learning, ensuring optimal policy preservation even when the feedback is inaccurate or adversarial. By formalizing this hybrid architecture as a Goal-Augmented Markov Decision Process, the study provides stronger guarantees than standard "LLM-as-reward" methods currently in use. The numerical verification on small MDPs demonstrates robustness against significant reward scaling errors, addressing critical stability concerns for integrating generative models into RL controllers.
Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents
arXiv:2608.18008v1 Announce Type: new Abstract: Combining large language models with reinforcement learning is increasingly explored, yet the theoretical status of LLM-derived reward signals is often left implicit. We formalize the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and show that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate. This guarantee is stronger than what general LLM-as-reward approaches provide. We verify the result numerically on a small MDP under four potential configurations, including an adversarial one scaled to twenty times the base reward magnitude.
Abstract:Combining large language models with reinforcement learning is increasingly explored, yet the theoretical status of LLM-derived reward signals is often left implicit. We formalize the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and show that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate. This guarantee is stronger than what general LLM-as-reward approaches provide. We verify the result numerically on a small MDP under four potential configurations, including an adversarial one scaled to twenty times the base reward magnitude.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2608.18008 [cs.LG]
(or arXiv:2608.18008v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.18008 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 200 cuvinte
Subiecte conexe
Assessing Information Ecosystems : How Governments Can Get Ahead of Hybrid Threats
carnegieendowment.org · 5 aug. 2026
Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback
arXiv · 30 iul. 2026
Scaling RL with verl on AMD Instinct MI355X: Async Walkthrough and Sync Benchmark
AMD · 18 aug. 2026
Frontier post-training recipe review with Finbarr Timbers
Interconnects · 16 iun. 2026