aiqbench analysis
This study investigates compute efficiency in Drifting Models by comparing resource allocation between optimizing parameters versus recomputing drift fields. The findings reveal a critical trade-off where spending budget on fresh field computations yields better FID scores at matched wall-clock time, challenging the assumption that deeper parameter fitting is always superior. This work provides actionable insights for scaling generative models by highlighting how different optimization goals diverge in this specific architecture.
Refresh or Realize? Compute Allocation in Drifting Models
arXiv:2609.32298v1 Announce Type: new Abstract: Drifting Models train a one-step generator by recomputing a finite-sample drift field at every iteration and taking an optimizer step toward the drifted target. The field says how generated samples should move, but the step is taken in parameters shared by all samples, so the motion the network actually makes need not match the motion it was given. This leaves a basic training question open: should extra compute go into fitting the current target more closely, or into recomputing the field? We study it on ImageNet 256x256. Holding the target fixed for k optimizer steps and measuring the realized displacement, we find that deeper fitting does bring the network closer to the frozen target, and that the number of steps needed before it makes any net progress drops from about sixteen early in training to one later on. When the extra steps come for free, k=2 also lowers FID. Once they are paid for, the result flips: at approximately matched measured wall-clock, spending the budget on fresh fields gives lower FID than deeper fitting, on both training seeds. The target itself shows why a fresh field is worth so much. Redrawing the finite support rotates its direction far more than a parameter update does (cosine ~0.3-0.6 against ~0.95), and a correction that is optimal in field space is not reliably better in FID than a parameter-free one. For Drifting, fitting each target well and spending compute well are different goals.
Title:Refresh or Realize? Compute Allocation in Drifting Models
Abstract:Drifting Models train a one-step generator by recomputing a finite-sample drift field at every iteration and taking an optimizer step toward the drifted target. The field says how generated samples should move, but the step is taken in parameters shared by all samples, so the motion the network actually makes need not match the motion it was given. This leaves a basic training question open: should extra compute go into fitting the current target more closely, or into recomputing the field? We study it on ImageNet 256x256. Holding the target fixed for k optimizer steps and measuring the realized displacement, we find that deeper fitting does bring the network closer to the frozen target, and that the number of steps needed before it makes any net progress drops from about sixteen early in training to one later on. When the extra steps come for free, k=2 also lowers FID. Once they are paid for, the result flips: at approximately matched measured wall-clock, spending the budget on fresh fields gives lower FID than deeper fitting, on both training seeds. The target itself shows why a fresh field is worth so much. Redrawing the finite support rotates its direction far more than a parameter update does (cosine ~0.3-0.6 against ~0.95), and a correction that is optimal in field space is not reliably better in FID than a parameter-free one. For Drifting, fitting each target well and spending compute well are different goals.
Machine Learning (cs.LG)
arXiv:2609.32298 [cs.LG]
(or arXiv:2609.32298v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.32298 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 342 words
Related coverage
Links #3: 2026/06 Part 1
LessWrong · Jun 15, 2026
The Long Road to the Same Answer: Cognitive Bias Under Escalating Reasoning Budgets in Large Language Models
arXiv · Oct 8, 2026
Differential Refresh Policies for Models Trained on Lagging Data Snapshots: From a Single-Age Equivalence Limit to an Optimal Per-Segment Allocation
arXiv · Oct 8, 2026
Inference meta-monitoring for Amazon SageMaker AI endpoints with Amazon Quick
AWS · Jul 30, 2026