aiqbench analysis
This research addresses a critical instability in off-policy reinforcement learning where the reuse of negative-advantage samples causes policies to drift indefinitely away from optimal actions. The authors propose Dynamic Remoteness-Aware Policy Optimization (DRPO), which selectively attenuates these remote updates while preserving local feedback, thereby restoring stable convergence and improving held-out rewards without discarding valuable historical data.
Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updates
arXiv:2602.10430v2 Announce Type: replace-cross Abstract: Off-policy policy optimization reuses historical behavior, including negative-advantage samples that suppress known failures. We show that repeated reuse can turn this useful signal into excessive repulsion: as the learner moves away from a historical negative action, subsequent updates make that action increasingly remote without necessarily reducing its update strength. Our aggregate theory characterizes the resulting transition from a stable displacement beyond the positive-only target to persistent drift and the loss of finite stable equilibria; controlled strength sweeps show that an intermediate displacement can improve held-out reward. The relevant learner-relative coordinate is squared standardized distance for Gaussian policies and surprisal for categorical policies. We introduce Dynamic Remoteness-Aware Policy Optimization (DRPO), which leaves the negative update unchanged in the near field and exponentially attenuates its remote tail. DRPO restores eventual inward Gaussian drift for every fixed finite negative-to-positive mass ratio, yields an explicit ultimate-bound radius, and changes categorical support suppression from exponential to polynomial probability decay. External diagnostics and controlled interventions isolate remoteness-dependent policy geometry as a source of negative-update amplification and show that selective tapering can remove its far-field effect without discarding useful local feedback.
Abstract:Off-policy policy optimization reuses historical behavior, including negative-advantage samples that suppress known failures. We show that repeated reuse can turn this useful signal into excessive repulsion: as the learner moves away from a historical negative action, subsequent updates make that action increasingly remote without necessarily reducing its update strength. Our aggregate theory characterizes the resulting transition from a stable displacement beyond the positive-only target to persistent drift and the loss of finite stable equilibria; controlled strength sweeps show that an intermediate displacement can improve held-out reward. The relevant learner-relative coordinate is squared standardized distance for Gaussian policies and surprisal for categorical policies. We introduce Dynamic Remoteness-Aware Policy Optimization (DRPO), which leaves the negative update unchanged in the near field and exponentially attenuates its remote tail. DRPO restores eventual inward Gaussian drift for every fixed finite negative-to-positive mass ratio, yields an explicit ultimate-bound radius, and changes categorical support suppression from exponential to polynomial probability decay. External diagnostics and controlled interventions isolate remoteness-dependent policy geometry as a source of negative-update amplification and show that selective tapering can remove its far-field effect without discarding useful local feedback.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2602.10430 [cs.LG]
(or arXiv:2602.10430v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.10430 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 279 words
Related coverage
October 2026 Patch Tuesday: 1,413 CVEs Include Three Zero-Days and 75 Critical Vulnerabilities
CrowdStrike Blog · Oct 13, 2026
Lock-In Risk Needs More Researchers; Here's Where to Start
LessWrong · Jun 17, 2026
Your prompt… plus everything you've ever (publicly) written
LessWrong · Oct 14, 2026
Against Export Controls (and China Threat Models)
LessWrong · Sep 24, 2026