aiqbench analysis
This research introduces TailRL, a reinforcement learning method that optimizes the probability of exceeding high reward thresholds rather than just maximizing average return. By focusing on the upper tails of the reward distribution, the approach ensures policies retain probability mass for rare but valuable outcomes, improving sample efficiency during inference. The technique integrates seamlessly into existing pipelines with minimal modifications to the advantage function, offering a practical upgrade for generative policy optimization.
Tail-Likelihood Reinforcement Learning
arXiv:2609.02987v1 Announce Type: new Abstract: Reinforcement learning typically optimizes average reward. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward while having very different chances of producing a rare but high-reward rollout. This matters as sampling increases during training and inference, since its benefit depends on retaining probability mass on high-reward outcomes. We propose to optimize this coverage directly. Rather than considering only expected reward, we consider all of its upper tails: for each reward threshold, how likely is the policy to exceed it? This turns a continuous reward into a family of binary success events. We introduce Tail-Likelihood Reinforcement Learning (TailRL), which maximizes the log-probability of exceeding a randomly chosen reward threshold. Its gradient gives more weight to rare, high-reward rollouts and can be interpreted as a mixture of Best-of-(k) gradients. TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL leverages rare high-reward training samples to avoid suboptimal solutions and yields models that benefit more from additional samples at inference time.
Authors:Shrinivas Ramasubramanian, Daman Arora, Fahim Tajwar, Guanning Zeng, Qingyang Wu, Zhongzhu Zhou, Chenfeng Xu, Haiwen Feng, Yuda Song, Aarti Singh, Ruslan Salakhutdinov, J. Andrew Bagnell, Jeff Schneider, Andrea Zanette
Abstract:Reinforcement learning typically optimizes average reward. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward while having very different chances of producing a rare but high-reward rollout. This matters as sampling increases during training and inference, since its benefit depends on retaining probability mass on high-reward outcomes. We propose to optimize this coverage directly. Rather than considering only expected reward, we consider all of its upper tails: for each reward threshold, how likely is the policy to exceed it? This turns a continuous reward into a family of binary success events. We introduce Tail-Likelihood Reinforcement Learning (TailRL), which maximizes the log-probability of exceeding a randomly chosen reward threshold. Its gradient gives more weight to rare, high-reward rollouts and can be interpreted as a mixture of Best-of-(k) gradients. TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL leverages rare high-reward training samples to avoid suboptimal solutions and yields models that benefit more from additional samples at inference time.
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2609.02987 [cs.LG]
(or arXiv:2609.02987v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.02987 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 236 words
Related coverage
Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization
arXiv · Oct 14, 2026
Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learning
arXiv · Jul 28, 2026
Why I care about imperfect-recall games
LessWrong · Oct 17, 2026
What We're Up Against: An AI Safety Crash Course
LessWrong · Sep 24, 2026