aiqbench analysis
This research identifies "zero collapse," a specific failure mode in policy-gradient methods where agents deteriorate into near-zero reward performance when operating in environments with discontinuous rewards, such as first-price auctions. The study demonstrates that this issue stems from severe continuation-value errors near decision thresholds and offers practical mitigations like adaptive update-magnitude control and learned baselines to stabilize learning. These findings are critical for developers deploying reinforcement learning systems in real-world scenarios involving sharp reward boundaries.
Zero Collapse: A Failure Mode of Policy Gradient Methods in Discontinuous Reward Environments
arXiv:2605.30896v2 Announce Type: replace Abstract: Policy-gradient learning can deteriorate even after substantial initial improvement. We study this behavior in a repeated first-price auction setting with a sharp threshold between winning and losing bids. We identify a failure mode, zero collapse, in which a policy shifts into persistently near-zero reward as informative outcomes become rarely sampled. Experiments with REINFORCE and a TD state-value actor-critic illustrate this failure and evaluate practical mitigations. Adaptive update-magnitude control and smooth-output parameterization support sustained REINFORCE learning, while a learned baseline substantially stabilizes the fixed-rate comparison. Some actor-critic runs still collapse under these interventions. A diagnostic near one observed collapse reveals severe continuation-value errors that reverse the relative desirability of actions across the winning threshold. These results illustrate how thresholded outcomes, policy updates, and inaccurate value estimates can contribute to zero collapse, and demonstrate practical mitigations in the studied setting.
Abstract:Policy-gradient learning can deteriorate even after substantial initial improvement. We study this behavior in a repeated first-price auction setting with a sharp threshold between winning and losing bids. We identify a failure mode, zero collapse, in which a policy shifts into persistently near-zero reward as informative outcomes become rarely sampled. Experiments with REINFORCE and a TD state-value actor-critic illustrate this failure and evaluate practical mitigations. Adaptive update-magnitude control and smooth-output parameterization support sustained REINFORCE learning, while a learned baseline substantially stabilizes the fixed-rate comparison. Some actor-critic runs still collapse under these interventions. A diagnostic near one observed collapse reveals severe continuation-value errors that reverse the relative desirability of actions across the winning threshold. These results illustrate how thresholded outcomes, policy updates, and inaccurate value estimates can contribute to zero collapse, and demonstrate practical mitigations in the studied setting.
Machine Learning (cs.LG)
arXiv:2605.30896 [cs.LG]
(or arXiv:2605.30896v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.30896 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 230 words
Related coverage
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026
Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimization
arXiv · Jun 9, 2026
When Partial Credit Opposes Completion
Snorkel AI Blog · Oct 15, 2026
The art and science of hyperparameter optimization on Amazon Nova Forge
AWS · Jun 2, 2026