analiza aiqbench
This paper establishes the theoretical minimax optimality of discrete diffusion models under sparsity assumptions, addressing a critical gap in their foundational theory compared to continuous counterparts. By demonstrating intrinsic adaptation to low-dimensional structures without requiring prior knowledge of sparsity parameters, the work validates discrete diffusion as a statistically efficient alternative to autoregressive methods for categorical data generation.
Discrete Diffusion Models are Minimax Optimal Under Sparsity
arXiv:2610.19020v1 Announce Type: cross Abstract: Discrete diffusion models have emerged as a leading alternative to autoregressive models for generating text and other categorical data, yet their theoretical foundations remain less developed than those of their continuous counterparts. For continuous diffusion models, a prominent line of theory suggests that these models can adapt to low-dimensional structure in the data, thereby improving statistical efficiency and generalization. In this work, we investigate the discrete analogue of this phenomenon by studying the statistical rates of discrete diffusion models under sparsity assumptions. By analyzing a nonparametric estimator derived from a regularized score-entropy objective, we establish minimax-optimal rates in the sparse regime. The proposed regularization can be viewed as an adaptation of label smoothing techniques introduced for classification, and does not require the knowledge of the sparsity parameter. This shows that discrete diffusion models can adapt intrinsically to sparsity.
Abstract:Discrete diffusion models have emerged as a leading alternative to autoregressive models for generating text and other categorical data, yet their theoretical foundations remain less developed than those of their continuous counterparts. For continuous diffusion models, a prominent line of theory suggests that these models can adapt to low-dimensional structure in the data, thereby improving statistical efficiency and generalization. In this work, we investigate the discrete analogue of this phenomenon by studying the statistical rates of discrete diffusion models under sparsity assumptions. By analyzing a nonparametric estimator derived from a regularized score-entropy objective, we establish minimax-optimal rates in the sparse regime. The proposed regularization can be viewed as an adaptation of label smoothing techniques introduced for classification, and does not require the knowledge of the sparsity parameter. This shows that discrete diffusion models can adapt intrinsically to sparsity.
Machine Learning (stat.ML); Machine Learning (cs.LG)
arXiv:2610.19020 [stat.ML]
(or arXiv:2610.19020v1 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2610.19020 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 235 cuvinte
Subiecte conexe
Uniform Discrete Diffusion Models are Minimax Optimal for Estimating Distributions with Small Effective Support Size
arXiv · 7 oct. 2026
Minimax Optimality of Score-Entropy Discrete Diffusion
arXiv · 24 aug. 2026
[AINews] Claude Opus 5.5, the new default model for AINews — and everybody cuts prices 40-50%
Latent Space · 23 sept. 2026
VSA: Accelerating Video Diffusion Inference with Sparse Attention on AMD GPUs
AMD · 4 aug. 2026