analiza aiqbench
This paper resolves fragmentation in discrete diffusion literature by proving that denoisers, bridge plug-ins, and concrete scores represent the same underlying object—the time-reversal of the noising process—expressed in different coordinates. The authors establish an "Oracle Distance" identity linking negative ELBO to data entropy and KL divergence, clarifying that bridge plug-ins specifically learn cavity distributions rather than standard denoisers. These theoretical insights provide a unified framework for interpreting training objectives and improving sampling quality across various noise kernels and logic tasks.
What Does a Discrete Diffusion Model Learn? One Reverse Process in Three Coordinates
arXiv:2607.05381v2 Announce Type: replace-cross Abstract: What does a discrete diffusion model actually learn? The literature trains denoisers, bridge plug-ins, and concrete scores, each deriving its own ELBO objective under different conventions. This fragmentation makes the object being learned unclear and ELBO values difficult to interpret and compare. All three methods learn the same object in different coordinates: the exact time-reversal of the noising process. We first prove the Oracle Distance identity: for any noising process, the negative ELBO is not just a bound on the likelihood; it equals the data entropy plus the path KL divergence from the reversed noising process to the learned one. Next, for token-factorizing noise, we show that denoiser, bridge plug-in, and concrete score are three coordinates of this reverse process, derive exact conversions between them, and identify the probability distribution learned by each. In particular, contrary to its standard interpretation, the bridge plug-in learns not a denoiser but the cavity distribution: the clean token given the noisy sequence excluding its own position. We confirm the theory by training all three coordinates with masked, uniform, and GIDD kernels on OpenWebText under a shared recipe, and by training uniform-diffusion solvers on four logic tasks. The learned coordinates agree after conversion, while confusing the cavity with the denoiser can severely degrade sampling quality. Together with exact boundary-term formulas, closed-form NELBO calibrations at initialization, and a clock interpretation of time schedules, these results provide a conceptual framework and a mechanical recipe for deriving, implementing, and comparing discrete diffusion processes.
Abstract:What does a discrete diffusion model actually learn? The literature trains denoisers, bridge plug-ins, and concrete scores, each deriving its own ELBO objective under different conventions. This fragmentation makes the object being learned unclear and ELBO values difficult to interpret and compare. All three methods learn the same object in different coordinates: the exact time-reversal of the noising process. We first prove the Oracle Distance identity: for any noising process, the negative ELBO is not just a bound on the likelihood; it equals the data entropy plus the path KL divergence from the reversed noising process to the learned one. Next, for token-factorizing noise, we show that denoiser, bridge plug-in, and concrete score are three coordinates of this reverse process, derive exact conversions between them, and identify the probability distribution learned by each. In particular, contrary to its standard interpretation, the bridge plug-in learns not a denoiser but the cavity distribution: the clean token given the noisy sequence excluding its own position. We confirm the theory by training all three coordinates with masked, uniform, and GIDD kernels on OpenWebText under a shared recipe, and by training uniform-diffusion solvers on four logic tasks. The learned coordinates agree after conversion, while confusing the cavity with the denoiser can severely degrade sampling quality. Together with exact boundary-term formulas, closed-form NELBO calibrations at initialization, and a clock interpretation of time schedules, these results provide a conceptual framework and a mechanical recipe for deriving, implementing, and comparing discrete diffusion processes.
48 pages, 10 figures. v2: revised and extended; accepted as an oral at the NeurIPS 2026 workshop Diffusion Language Models: Foundations, Efficiency, and Reasoning
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Information Theory (cs.IT); Machine Learning (stat.ML)
arXiv:2607.05381 [cs.LG]
(or arXiv:2607.05381v2 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 293 cuvinte
Subiecte conexe
Spectral Feedback for Test-Time Alignment of Protein Diffusion Models
arXiv · 28 sept. 2026
From Score Learning to Discretized Sampling: An End-to-End Generalization Analysis of Diffusion Models
arXiv · 28 iul. 2026
Foundations of Diffusion Models in General State Spaces: A Self-Contained Introduction
arXiv · 20 aug. 2026
Connections between the F\"ollmer process and the denoising diffusion probabilistic model
arXiv · 3 sept. 2026