aiqbench analysis
This research introduces Experiential Learning (EL), which transforms an LLM judge into a coach to distill rich, transferable feedback rather than relying on compressed scalar rewards. By preserving fine-grained preferences during post-training, the method significantly improves generalization and mitigates reward hacking on open-ended tasks where verification is difficult. This approach represents a meaningful advancement in reinforcement learning strategies for complex AI agents facing non-verifiable objectives.
LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks
arXiv:2607.18110v1 Announce Type: cross Abstract: Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.
Abstract:Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2607.18110 [cs.LG]
(or arXiv:2607.18110v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.18110 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 241 words
Related coverage
Microsoft’s AI chief says superintelligence is near, but won’t take your job
The Verge · Jun 8, 2026
Learning to Coach for Experiential Learning
arXiv · Sep 15, 2026
Current AIs out-persuade professionals in lab settings but (probably) not in the world
LessWrong · Oct 5, 2026
How funny are the frontier AI models?
LessWrong · Oct 6, 2026