aiqbench analysis
Residual reinforcement learning for a frozen pretrained generative robot policy: the paper proposes CANDO, which combines noise-space adjustments (no fixed latent feasible regions) with a local action-space correction. The authors report 76.1 NAUC and a 91.4% final success rate, 6.6 NAUC and 4.1 points above their best single-space baseline, but these are self-reported preprint results, and the benchmarks, baselines, and robustness across tasks still need independent checking. For ML readers, the practical question is whether the hybrid design generalizes beyond the reported robot tasks.
CANDO: Conservative Action-Noise Directional Optimization for Residual Reinforcement Learning
arXiv:2611.04543v1 Announce Type: new Abstract: Pretrained generative policies provide strong behavior priors for robot control, but adapting them efficiently with online reinforcement learning remains difficult. A central challenge is how to improve a frozen generative policy without destroying the structure learned from offline demonstrations. Noise-space residual methods address this by modifying the policy's input noise, but existing noise-space methods typically regulate adaptation through fixed latent feasible regions, which may exclude valuable noise edits. In contrast, action-space residual methods are more sample-efficient, but their efficiency can still be further improved. We propose Conservative Noise Directional Optimization (CNDO), a noise-space residual RL method that improves a frozen generative policy by learning state- and noise-conditioned directions around Gaussian latent samples. CNDO removes predefined latent feasible regions and instead regulates noise-space adaptation through the soft conservative constraint, which consists of conservative value estimation and gated execution. We further introduce Conservative Action-Noise Directional Optimization (CANDO), which adds an action-space residual, combining structured latent-space improvement with local action correction. Experiments show that, once CNDO narrows the gap between noise-space and action-space residual learning, CANDO further improves over single-space optimization, achieving 76.1 NAUC and a 91.4% final success rate, surpassing the best single-space results by 6.6 NAUC and 4.1% success rate. More demos and info are available on https://cando-rl.github.io/.
Abstract:Pretrained generative policies provide strong behavior priors for robot control, but adapting them efficiently with online reinforcement learning remains difficult. A central challenge is how to improve a frozen generative policy without destroying the structure learned from offline demonstrations. Noise-space residual methods address this by modifying the policy's input noise, but existing noise-space methods typically regulate adaptation through fixed latent feasible regions, which may exclude valuable noise edits. In contrast, action-space residual methods are more sample-efficient, but their efficiency can still be further improved. We propose Conservative Noise Directional Optimization (CNDO), a noise-space residual RL method that improves a frozen generative policy by learning state- and noise-conditioned directions around Gaussian latent samples. CNDO removes predefined latent feasible regions and instead regulates noise-space adaptation through the soft conservative constraint, which consists of conservative value estimation and gated execution. We further introduce Conservative Action-Noise Directional Optimization (CANDO), which adds an action-space residual, combining structured latent-space improvement with local action correction. Experiments show that, once CNDO narrows the gap between noise-space and action-space residual learning, CANDO further improves over single-space optimization, achieving 76.1 NAUC and a 91.4% final success rate, surpassing the best single-space results by 6.6 NAUC and 4.1% success rate. More demos and info are available on this https URL.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2611.04543 [cs.LG]
(or arXiv:2611.04543v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2611.04543 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 306 words
Related coverage
Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training
arXiv · Oct 5, 2026
Web Agent Agentic Reinforcement Learning Decision Model Under Multi-Cost and Failure Risk
arXiv · Oct 21, 2026
Counterfactual Transport Flows for Offline Conservative Trajectory Refinement
arXiv · Jun 9, 2026
Residual Visual Credit Optimization: Conserved Evidence Routing for Multimodal Reinforcement Learning
arXiv · Oct 6, 2026