analiza aiqbench
This research introduces a novel reinforcement learning framework that derives policy updates solely from pairwise outcome comparisons, eliminating the need for explicit scalar reward functions or pre-trained reward models. The study demonstrates that this comparison-based approach maintains performance comparable to standard scalar return methods while offering improved robustness against label noise and better alignment with LLM judges in complex tasks.
Ordinal Credit Assignment: Multi-Step Reinforcement Learning from Comparisons Alone
arXiv:2605.06375v2 Announce Type: replace-cross Abstract: Policy-gradient methods need a scalar reward, yet for many tasks no formula captures what we want, while judging which of two outcomes is better is easy. We study reinforcement learning that receives only such comparisons - no reward function, no reward model - in multi-step tasks. Groups of trajectories start from a shared state, and each action is credited by how its reward-to-go compares with those of other trajectories at the same time step. We show the expected update is policy gradient with reward-to-go replaced by its win rate against the population still running at that time, weighted by the chance an opponent is still running; in deterministic environments the update ranks actions by true value and the optimal policy is stable, whereas under stochastic outcomes it prefers actions that win more often rather than those with higher mean. Label noise near ties or from position bias leaves the expected update unchanged; only noise that also flips clear comparisons halves the signal. A critic trained from the same comparisons removes the state confound without changing the expected update. In a pre-registered study over six environments with ten seeds per claim, comparisons cost no detectable performance relative to scalar returns at the same granularity, learning a reward model from the same comparisons is not better, and ten percent of comparisons suffice when spent on a few time points with all pairs. With real LLM judges, comparisons are 7-10 points more accurate than pointwise scores, and the gap grows with the stronger judge. Trained end-to-end with an LLM judge on a requirement the hand-written reward omits, policies satisfy it in all five seeds but land far less often than with oracle comparisons, because the judge is near chance on the near-tied failures dominating early training. Comparisons do not close the gap to PPO with scalar rewards on HalfCheetah.
Abstract:Policy-gradient methods need a scalar reward, yet for many tasks no formula captures what we want, while judging which of two outcomes is better is easy. We study reinforcement learning that receives only such comparisons - no reward function, no reward model - in multi-step tasks. Groups of trajectories start from a shared state, and each action is credited by how its reward-to-go compares with those of other trajectories at the same time step. We show the expected update is policy gradient with reward-to-go replaced by its win rate against the population still running at that time, weighted by the chance an opponent is still running; in deterministic environments the update ranks actions by true value and the optimal policy is stable, whereas under stochastic outcomes it prefers actions that win more often rather than those with higher mean. Label noise near ties or from position bias leaves the expected update unchanged; only noise that also flips clear comparisons halves the signal. A critic trained from the same comparisons removes the state confound without changing the expected update. In a pre-registered study over six environments with ten seeds per claim, comparisons cost no detectable performance relative to scalar returns at the same granularity, learning a reward model from the same comparisons is not better, and ten percent of comparisons suffice when spent on a few time points with all pairs. With real LLM judges, comparisons are 7-10 points more accurate than pointwise scores, and the gap grows with the stronger judge. Trained end-to-end with an LLM judge on a requirement the hand-written reward omits, policies satisfy it in all five seeds but land far less often than with oracle comparisons, because the judge is near chance on the near-tied failures dominating early training. Comparisons do not close the gap to PPO with scalar rewards on HalfCheetah.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Statistics Theory (math.ST)
arXiv:2605.06375 [cs.LG]
(or arXiv:2605.06375v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.06375 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 404 cuvinte
Subiecte conexe
Comprehensive Comparisons of Reinforcement Learning Algorithms in Autonomous Driving
arXiv · 15 oct. 2026
A Quantitative Comparison of Centralised and Distributed Reinforcement Learning-Based Control for Soft Robotic Arms
arXiv · 4 sept. 2026
GeoRouteNet: A Geometry-Aware Non-Autoregressive Neural Solver for the Euclidean Traveling Salesman Problem
arXiv · 20 iul. 2026
Fine-Tuning Small Reasoning Models for Quantum Field Theory
arXiv · 22 oct. 2026