aiqbench analysis
This research investigates the paradoxical behavior of diffusion models, demonstrating that while they generalize well in practice, an optimal model would theoretically fully memorize training data and fail to generalize. The authors analyze how progressive overfitting creates a generalization gap at intermediate noise levels, revealing that model error suppresses exact recall to produce smooth, generalizing flow fields. These findings clarify the distinct mechanisms of structural memorization within denoising objectives and inference trajectories.
Beyond Sample Copying: Structural Memorization in Diffusion Models
arXiv:2603.13419v3 Announce Type: replace Abstract: Diffusion models generalize well in practice. Paradoxically, an optimal diffusion model fully memorizes the training data and therefore fails to generalize, raising the question of what induces generalization in a real diffusion model. We show that diffusion models progressively overfit the denoising training objective, creating a generalization gap between validation and training performance at intermediate noise levels. In a fully analytic 2D toy model with a controlled denoising error, we trace this gap to the interaction between model error and the density of the data distribution's support. The optimal denoising flow field localizes sharply around individual training points, whereas model error suppresses exact recall of training points, yielding a smooth, generalizing flow field. Finally, we examine how training-time overfitting manifests along inference trajectories. We find that predictions made from intermediate trajectory states occupy a distinct feature-space regime from predictions made from noised training and validation images. As training progresses and model size increases, these predictions develop greater relative affinity to the training data, despite the absolute similarity to the validation data not decreasing. Together, these findings show that the denoising objective and inference trajectories express structural memorization differently.
Abstract:Diffusion models generalize well in practice. Paradoxically, an optimal diffusion model fully memorizes the training data and therefore fails to generalize, raising the question of what induces generalization in a real diffusion model. We show that diffusion models progressively overfit the denoising training objective, creating a generalization gap between validation and training performance at intermediate noise levels. In a fully analytic 2D toy model with a controlled denoising error, we trace this gap to the interaction between model error and the density of the data distribution's support. The optimal denoising flow field localizes sharply around individual training points, whereas model error suppresses exact recall of training points, yielding a smooth, generalizing flow field. Finally, we examine how training-time overfitting manifests along inference trajectories. We find that predictions made from intermediate trajectory states occupy a distinct feature-space regime from predictions made from noised training and validation images. As training progresses and model size increases, these predictions develop greater relative affinity to the training data, despite the absolute similarity to the validation data not decreasing. Together, these findings show that the denoising objective and inference trajectories express structural memorization differently.
Machine Learning (cs.LG)
arXiv:2603.13419 [cs.LG]
(or arXiv:2603.13419v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2603.13419 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 280 words
Related coverage
First Learn, Then Memorize: The Spectral Bias of Diffusion Models
arXiv · Sep 30, 2026
Towards demystifying the creativity of diffusion models
Google Research · Jul 15, 2026
Early Signatures of Memorization in Diffusion Models via Basin Geometry and Cyclic Denoising
arXiv · Oct 9, 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026