analiza aiqbench
This paper introduces ZeNO, a gradient-free optimization framework designed to align generative models like diffusion and flow models without requiring backpropagation through the reward pipeline. By formulating noise-space optimization as a path-integral control problem, it enables effective inference-time scaling for deterministic generators and complex tasks such as protein structure generation where traditional differentiable methods fail. The work advances the theoretical and practical boundaries of reward alignment in non-differentiable settings, offering a robust solution for diverse generator architectures.
Gradient-Free Noise Optimization for Reward Alignment in Generative Models
arXiv:2605.11347v3 Announce Type: replace Abstract: Existing reward alignment methods for diffusion and flow models rely on multi-step stochastic trajectories, making them difficult to extend to deterministic generators. A natural alternative is noise-space optimization, but existing approaches require backpropagation through the generator and reward pipeline, limiting applicability to differentiable settings. To address this, here we present ZeNO (Zeroth-order Noise Optimization), a gradient-free framework that formulates noise optimization as a path-integral control problem, estimable from zeroth-order reward evaluations. When instantiated with an Ornstein--Uhlenbeck reference process and a one-step receding horizon, the repeated update connects to Langevin dynamics targeting a reward-tilted distribution. ZeNO enables effective inference-time scaling and demonstrates strong performance across diverse generators and reward functions, including a protein structure generation task where backpropagation is infeasible.
Abstract:Existing reward alignment methods for diffusion and flow models rely on multi-step stochastic trajectories, making them difficult to extend to deterministic generators. A natural alternative is noise-space optimization, but existing approaches require backpropagation through the generator and reward pipeline, limiting applicability to differentiable settings. To address this, here we present ZeNO (Zeroth-order Noise Optimization), a gradient-free framework that formulates noise optimization as a path-integral control problem, estimable from zeroth-order reward evaluations. When instantiated with an Ornstein--Uhlenbeck reference process and a one-step receding horizon, the repeated update connects to Langevin dynamics targeting a reward-tilted distribution. ZeNO enables effective inference-time scaling and demonstrates strong performance across diverse generators and reward functions, including a protein structure generation task where backpropagation is infeasible.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2605.11347 [cs.LG]
(or arXiv:2605.11347v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.11347 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 220 cuvinte