analiza aiqbench
This research reveals that diffusion models exhibit a "simplicity bias," prioritizing linear sample complexity for pair-wise statistics while requiring cubic complexity to capture higher-order correlations like the fourth cumulant. The authors introduce the diffusion information exponent as a scalar invariant that quantifies this learning hierarchy and demonstrates that correlated latent structures can linearize the learning of complex statistics. For practitioners, this implies that achieving high-fidelity generation of complex distributions depends critically on whether the underlying data structure allows for efficient transfer of learned statistics from simple to complex correlations.
A theory of learning data statistics in diffusion models, from easy to hard
arXiv:2603.12901v2 Announce Type: replace-cross Abstract: While diffusion models have emerged as a powerful class of generative models, their learning dynamics remain poorly understood. We address this issue first by empirically showing that standard diffusion models trained on natural images exhibit a distributional simplicity bias, learning simple, pair-wise input statistics before specializing to higher-order correlations. We reproduce this behaviour in simple denoisers trained on a minimal data model, the mixed cumulant model, where we precisely control both pair-wise and higher-order correlations of the inputs. We identify a scalar invariant of the model that governs the sample complexity of learning pair-wise and higher-order correlations that we call the diffusion information exponent, in analogy to related invariants in different learning paradigms. Using this invariant, we prove that the denoiser learns simple, pair-wise statistics of the inputs at linear sample complexity, while more complex higher-order statistics, such as the fourth cumulant, require at least cubic sample complexity. We also prove that the sample complexity of learning the fourth cumulant is linear if pair-wise and higher-order statistics share a correlated latent structure. Our work describes a key mechanism for how diffusion models can learn distributions of increasing complexity.
Abstract:While diffusion models have emerged as a powerful class of generative models, their learning dynamics remain poorly understood. We address this issue first by empirically showing that standard diffusion models trained on natural images exhibit a distributional simplicity bias, learning simple, pair-wise input statistics before specializing to higher-order correlations. We reproduce this behaviour in simple denoisers trained on a minimal data model, the mixed cumulant model, where we precisely control both pair-wise and higher-order correlations of the inputs. We identify a scalar invariant of the model that governs the sample complexity of learning pair-wise and higher-order correlations that we call the diffusion information exponent, in analogy to related invariants in different learning paradigms. Using this invariant, we prove that the denoiser learns simple, pair-wise statistics of the inputs at linear sample complexity, while more complex higher-order statistics, such as the fourth cumulant, require at least cubic sample complexity. We also prove that the sample complexity of learning the fourth cumulant is linear if pair-wise and higher-order statistics share a correlated latent structure. Our work describes a key mechanism for how diffusion models can learn distributions of increasing complexity.
Machine Learning (stat.ML); Disordered Systems and Neural Networks (cond-mat.dis-nn); Information Theory (cs.IT); Machine Learning (cs.LG)
arXiv:2603.12901 [stat.ML]
(or arXiv:2603.12901v2 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2603.12901 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 292 cuvinte
Subiecte conexe
Ergodic Control and Controlled Diffusion for Robot Learning: Review and Tutorial
arXiv · 15 sept. 2026
Uniform Discrete Diffusion Models are Minimax Optimal for Estimating Distributions with Small Effective Support Size
arXiv · 7 oct. 2026
Generalizable dose denoising using diffusion transformer framework for pencil beam scanning proton therapy
arXiv · 31 oct. 2026
On the minimax optimality of Flow Matching through the connection to kernel density estimation
arXiv · 30 oct. 2026