aiqbench analysis
This paper provides a theoretical framework analyzing when trajectory-based training updates effectively reduce decision risk in models evaluated by downstream rewards, addressing the common mismatch between surrogate losses and actual utility gains. By decomposing transfer into gradient misalignment and curvature terms, it establishes precise conditions for first-order improvement via positive collinearity of accessible gradients while bounding regret through calibration gaps. Controlled experiments on gridworlds and LLM post-training confirm these predictions, offering rigorous guidance for optimizing training trajectories in complex reinforcement learning scenarios.
When Do Surrogate Updates Improve Decisions? A Local Theory of Trajectory-Wise Transfer
arXiv:2608.01130v1 Announce Type: new Abstract: A broad range of models face the mismatch where they are updated through trajectory losses but are evaluated by downstream task reward. Here, a trajectory is a training instance that induces a surrogate loss whose reduction might not track the model's decision utility update. Theoretically, we ask when one step of trajectory training reduces both population surrogate loss and decision risk, and how transfer accumulates along repeated updates. To formalize this, we first fix a checkpoint and a restricted update space, and define the reductions in population surrogate risk and decision risk induced by a trajectory as its learnability and decision utility, respectively. On this basis, our theory yields four main results. First, a one-step transfer bound separates their discrepancy into first-order gradient misalignment after nonnegative calibration and second-order curvature; and a pathwise extension accumulates the same terms over repeated updates. Second, when the accessible surrogate gradient is nonzero, universal first-order transfer over every accessible direction holds exactly when the accessible surrogate and decision gradients are positively collinear. Third, the calibration gap bounds the decision regret of learnability-based trajectory selection, while a candidate-difference refinement tightens this guarantee by retaining only directions that affect pairwise rankings. Finally, we establish an approximation--calibration trade-off across nested update spaces. Controlled gridworld and LLM post-training experiments yield results consistent with our predictions.
Title:When Do Surrogate Updates Improve Decisions? A Local Theory of Trajectory-Wise Transfer
Abstract:A broad range of models face the mismatch where they are updated through trajectory losses but are evaluated by downstream task reward. Here, a trajectory is a training instance that induces a surrogate loss whose reduction might not track the model's decision utility update. Theoretically, we ask when one step of trajectory training reduces both population surrogate loss and decision risk, and how transfer accumulates along repeated updates. To formalize this, we first fix a checkpoint and a restricted update space, and define the reductions in population surrogate risk and decision risk induced by a trajectory as its learnability and decision utility, respectively. On this basis, our theory yields four main results. First, a one-step transfer bound separates their discrepancy into first-order gradient misalignment after nonnegative calibration and second-order curvature; and a pathwise extension accumulates the same terms over repeated updates. Second, when the accessible surrogate gradient is nonzero, universal first-order transfer over every accessible direction holds exactly when the accessible surrogate and decision gradients are positively collinear. Third, the calibration gap bounds the decision regret of learnability-based trajectory selection, while a candidate-difference refinement tightens this guarantee by retaining only directions that affect pairwise rankings. Finally, we establish an approximation--calibration trade-off across nested update spaces. Controlled gridworld and LLM post-training experiments yield results consistent with our predictions.
22 pages in total, including references and appendices; 3 composite figures (9 panels) and 4 tables. Code is available at this https URL
Machine Learning (cs.LG)
arXiv:2608.01130 [cs.LG]
(or arXiv:2608.01130v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.01130 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 346 words
Related coverage
[AINews] not much happened today
Latent Space · Oct 3, 2026
The Entangled Dimensions of Decision Theory
LessWrong · Jul 30, 2026
[AINews] not much happened today
Latent Space · Sep 18, 2026
Learned Parallel Bit-Flipping Sequential Belief Propagation Decoding of Quantum LDPC Codes
arXiv · Oct 2, 2026