aiqbench analysis
This research proposes MaxRL, a new objective function that reframes reinforcement learning as a maximum likelihood problem to address the non-differentiability of standard reward-based approaches. By interpolating between expected-reward RL and true likelihood estimation, the method achieves significant efficiency gains in test-time scaling without requiring architectural changes beyond a single line of code. The findings suggest a fundamental shift in how sample-based objectives are formulated for training models where feedback is limited to binary outcomes.
Maximum Likelihood Reinforcement Learning
arXiv:2602.02710v2 Announce Type: replace Abstract: Reinforcement learning (RL) is the method of choice for training models in setups where the objective function can only be evaluated by sampling from the model. Our key observation is that when the feedback is terminal and binary, models implicitly induce a likelihood over correct rollouts. Maximum likelihood would be the natural framework in such settings, but RL is used instead as a workaround to the non-differentiability. We prove that the standard, expected-reward RL formulation is only a first-order approximation of the likelihood. To remedy this mismatch, we introduce Maximum Likelihood Reinforcement Learning (MaxRL), a compute-indexed family of sample-based objectives that interpolate between expected-reward RL and maximum likelihood as sampling compute is scaled. The resulting objective is a one-line change to standard RL implementations. MaxRL Pareto-dominates existing methods in all tested models and tasks, achieves up to $20\times$ gains in test-time scaling efficiency over GRPO, and scales more favorably with additional training data and compute.
Abstract:Reinforcement learning (RL) is the method of choice for training models in setups where the objective function can only be evaluated by sampling from the model. Our key observation is that when the feedback is terminal and binary, models implicitly induce a likelihood over correct rollouts. Maximum likelihood would be the natural framework in such settings, but RL is used instead as a workaround to the non-differentiability. We prove that the standard, expected-reward RL formulation is only a first-order approximation of the likelihood. To remedy this mismatch, we introduce Maximum Likelihood Reinforcement Learning (MaxRL), a compute-indexed family of sample-based objectives that interpolate between expected-reward RL and maximum likelihood as sampling compute is scaled. The resulting objective is a one-line change to standard RL implementations. MaxRL Pareto-dominates existing methods in all tested models and tasks, achieves up to $20\times$ gains in test-time scaling efficiency over GRPO, and scales more favorably with additional training data and compute.
ICML 2026. Project website: this https URL
Machine Learning (cs.LG)
arXiv:2602.02710 [cs.LG]
(or arXiv:2602.02710v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.02710 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 254 words
Related coverage
Exploding variance of means of exponentials: least-squares to the rescue – Machine Learning Research Blog
francisbach.com · Sep 27, 2026
SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning
arXiv · Sep 30, 2026
Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward
arXiv · Jun 16, 2026
Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
arXiv · Sep 11, 2026