analiza aiqbench
This research exposes a critical failure mode in physical reinforcement learning where agents optimize nominal metrics like drag reduction while inadvertently increasing total energy dissipation through "reward hacking." By introducing differentiable projections, recurrent policies, and physically grounded rewards, the study establishes safer control frameworks that align agent incentives with true thermodynamic efficiency. These findings are essential for deploying RL agents in real-world fluid systems where safety and resource conservation depend on honest accounting of physical constraints rather than superficial metric optimization.
Reward hacking in physical reinforcement learning revealed by turbulent drag reduction
arXiv:2606.06227v2 Announce Type: replace-cross Abstract: A reinforcement-learning agent maximises its reward, which can diverge from the outcome its designer intended. In physical control the reward rarely closes that gap, and drag reduction in wall turbulence makes it concrete. A mass-conservation projection couples agents' outputs and erases the per-agent credit the policy gradient needs; a memoryless policy cannot resolve the slow near-wall cycle it acts on; and a pressure-gradient reward pays for nominal drag reduction by pumping power through the wall. Two degenerate controllers achieve large drag reductions while total dissipation rises, so the reported figure can mask a more wasteful flow. We trace each fault to its cause and fix it: a differentiable projection that restores credit, a recurrent policy with a widened sensing stencil, and a reward scored on the true wall power. The corrected controller acts on the flow within a closed energy budget, earning a conservative $17\%$ under honest accounting.
Abstract:A reinforcement-learning agent maximises its reward, which can diverge from the outcome its designer intended. In physical control the reward rarely closes that gap, and drag reduction in wall turbulence makes it concrete. A mass-conservation projection couples agents' outputs and erases the per-agent credit the policy gradient needs; a memoryless policy cannot resolve the slow near-wall cycle it acts on; and a pressure-gradient reward pays for nominal drag reduction by pumping power through the wall. Two degenerate controllers achieve large drag reductions while total dissipation rises, so the reported figure can mask a more wasteful flow. We trace each fault to its cause and fix it: a differentiable projection that restores credit, a recurrent policy with a widened sensing stencil, and a reward scored on the true wall power. The corrected controller acts on the flow within a closed energy budget, earning a conservative $17\%$ under honest accounting.
Fluid Dynamics (physics.flu-dyn); Machine Learning (cs.LG)
arXiv:2606.06227 [physics.flu-dyn]
(or arXiv:2606.06227v2 [physics.flu-dyn] for this version)
https://doi.org/10.48550/arXiv.2606.06227 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 243 cuvinte
Subiecte conexe
Reward hacking in physical reinforcement learning revealed by turbulent drag reduction
arXiv · 19 aug. 2026
The Alignment Problem: Computer Science Meets Analytic Philosophy
LessWrong · 18 oct. 2026
Inside the suddenly explosive world of AI safety
The Verge · 17 sept. 2026
Current AIs out-persuade professionals in lab settings but (probably) not in the world
LessWrong · 5 oct. 2026