Fine-Grained Caching for Diffusion Transformers with Few Calibration Conditions

arXiv:2512.05134v2 Announce Type: replace-cross Abstract: Diffusion transformers require repeated denoiser evaluations, making image and video generation computationally expensive. We propose a training-free framework that uses a few calibration conditions to construct a fixed, fine-grained module-reuse schedule without schedule search. The design is motivated by an empirical effect along deterministic sampling trajectories: with initial noise fixed, condition-dependent deviations in several module outputs evolve similarly across adjacent steps, so temporal differencing attenuates much of their variation. We term this effect Conditional Common-Mode Rejection (CCMR). Motivated by this observation, we rank timestep--layer--module locations using normalized ratios of consecutive feature displacements and construct a Cache Book through two-stage calibration. The second stage updates a scorer-only reference with each provisional bit after recording its indexed score, while retaining full-compute trajectories. At inference, the fixed schedule requires no per-input policy estimation and can be nested within fixed step-wise cache policies. Module caching alone achieves $1.66$--$1.72\times$ speedup on the evaluated image models and $1.54$--$1.68\times$ on the video models. Combining it with MagCache or SeaCache provides further acceleration at a model-dependent cost in paired fidelity.

Abstract:Diffusion transformers require repeated denoiser evaluations, making image and video generation computationally expensive. We propose a training-free framework that uses a few calibration conditions to construct a fixed, fine-grained module-reuse schedule without schedule search. The design is motivated by an empirical effect along deterministic sampling trajectories: with initial noise fixed, condition-dependent deviations in several module outputs evolve similarly across adjacent steps, so temporal differencing attenuates much of their variation. We term this effect Conditional Common-Mode Rejection (CCMR). Motivated by this observation, we rank timestep--layer--module locations using normalized ratios of consecutive feature displacements and construct a Cache Book through two-stage calibration. The second stage updates a scorer-only reference with each provisional bit after recording its indexed score, while retaining full-compute trajectories. At inference, the fixed schedule requires no per-input policy estimation and can be nested within fixed step-wise cache policies. Module caching alone achieves $1.66$--$1.72\times$ speedup on the evaluated image models and $1.54$--$1.68\times$ on the video models. Combining it with MagCache or SeaCache provides further acceleration at a model-dependent cost in paired fidelity.

Computer Vision and Pattern Recognition (cs.CV); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)

arXiv:2512.05134 [cs.CV]

(or arXiv:2512.05134v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2512.05134 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Fine-Grained Caching for Diffusion Transformers with Few Calibration Conditions

Related coverage