aiqbench analysis
This research addresses a critical inefficiency in training models via black-box oracles, demonstrating that standard action-value functions often fail to achieve variance optimality due to shared parameter interactions. The authors propose a simple, cost-free unbiased correction derived from an exact variance decomposition that significantly reduces gradient noise and accelerates learning across diverse tasks. For practitioners relying on human-in-the-loop systems or external solvers, this method offers a direct path to more stable and efficient optimization without additional data collection costs.
Variance-Optimal Control Variates for Learning with Black-box Feedback
arXiv:2610.04766v1 Announce Type: new Abstract: Modern models increasingly learn through black-box oracles such as humans, optimization solvers, and external tools that provide feedback without exposing their internal mechanisms. A common remedy is to learn an (action-)value function as a control variate. In this paper, we first observe that even an exact action-value function can be arbitrarily far from variance-optimal. We show that this gap arises because the value function minimizes the noise in each action's own gradient term, while an action can still affect the rest of the gradient estimator through shared parameters. A simple unbiased correction, at no extra oracle cost, can still reduce its variance by an arbitrarily large factor. Motivated by this, we then prove that the residual variance can be decomposed exactly by actions with no cross terms. This decomposition yields a closed-form variance-minimizing correction for neural-network parameters, which can be computed by a simple projection. Empirically, our correction consistently reduces the variance left by the value function and improves learning across all tasks. The source code for all experiments is available at https://github.com/Zihao-Kevin/black_box_opt.
Abstract:Modern models increasingly learn through black-box oracles such as humans, optimization solvers, and external tools that provide feedback without exposing their internal mechanisms. A common remedy is to learn an (action-)value function as a control variate. In this paper, we first observe that even an exact action-value function can be arbitrarily far from variance-optimal. We show that this gap arises because the value function minimizes the noise in each action's own gradient term, while an action can still affect the rest of the gradient estimator through shared parameters. A simple unbiased correction, at no extra oracle cost, can still reduce its variance by an arbitrarily large factor. Motivated by this, we then prove that the residual variance can be decomposed exactly by actions with no cross terms. This decomposition yields a closed-form variance-minimizing correction for neural-network parameters, which can be computed by a simple projection. Empirically, our correction consistently reduces the variance left by the value function and improves learning across all tasks. The source code for all experiments is available at this https URL.
Machine Learning (cs.LG)
arXiv:2610.04766 [cs.LG]
(or arXiv:2610.04766v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.04766 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 269 words
Related coverage
Building and evaluating model diffing agents
AI Alignment Forum · Jun 12, 2026
Geometric Distributional Control: Learning Progress with Partial Structural Knowledge
arXiv · Sep 9, 2026
Custom reward functions for multi-turn reinforcement learning with Amazon Nova Forge
AWS · Aug 14, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026