analiza aiqbench
This research introduces Systematic Hazard Sampling (SHS), a training-free inference technique designed to minimize variance in discrete diffusion and flow models. By replacing stochastic Bernoulli decisions with systematic hazard thresholds, SHS directly addresses common failure modes like under-editing and over-editing without altering the underlying model architecture or requiring additional hyperparameters. The method demonstrates consistent quality improvements across various model sizes by optimizing the sampling trajectory itself rather than retraining the weights.
Systematic Hazard Sampling: Minimal-Variance Inference for Discrete Diffusion and Flow Models
arXiv:2601.02799v3 Announce Type: replace-cross Abstract: Uniform-noise discrete diffusion and flow models generate sequences non-autoregressively through iterative, context-dependent token replacements. However, these models are typically formulated as time-inhomogeneous continuous- or discrete-time Markov chains (CTMC/DTMC), sampled using independent Bernoulli change decisions per discretization step. This induces Poisson-binomial variance in per-position jump counts that grows with the number of required edits, leading to the common under-editing (residual noise) and over-editing (cascading substitutions) failure modes that degrade sample quality. We identify this sampler-induced variance as an orthogonal source of degradation, distinct from model-side errors and addressable purely at inference time. We propose Systematic Hazard Sampling (SHS), a training-free, drop-in, and hyperparameter-free inference principle for any sampler that admits a stay-vs.-replace decomposition. SHS models per-token edits as events driven by cumulative hazard (CTMC) or jump mass (DTMC) and triggers an edit whenever this quantity exceeds unit-spaced thresholds with a single random phase per position. For any fixed cumulative mass, this preserves the expected jump count while achieving the minimum conditional variance possible among unbiased integer estimators (at most 1/4), without altering per-jump destination sampling. Experiments on four uniform-noise discrete diffusion and flow language models spanning ~110M to ~3B parameters show that SHS consistently improves sample quality across numbers of function evaluations (NFE). Code is available at https://github.com/Jang-seunghwan/Systematic-Hazard-Sampling.
Abstract:Uniform-noise discrete diffusion and flow models generate sequences non-autoregressively through iterative, context-dependent token replacements. However, these models are typically formulated as time-inhomogeneous continuous- or discrete-time Markov chains (CTMC/DTMC), sampled using independent Bernoulli change decisions per discretization step. This induces Poisson-binomial variance in per-position jump counts that grows with the number of required edits, leading to the common under-editing (residual noise) and over-editing (cascading substitutions) failure modes that degrade sample quality. We identify this sampler-induced variance as an orthogonal source of degradation, distinct from model-side errors and addressable purely at inference time. We propose Systematic Hazard Sampling (SHS), a training-free, drop-in, and hyperparameter-free inference principle for any sampler that admits a stay-vs.-replace decomposition. SHS models per-token edits as events driven by cumulative hazard (CTMC) or jump mass (DTMC) and triggers an edit whenever this quantity exceeds unit-spaced thresholds with a single random phase per position. For any fixed cumulative mass, this preserves the expected jump count while achieving the minimum conditional variance possible among unbiased integer estimators (at most 1/4), without altering per-jump destination sampling. Experiments on four uniform-noise discrete diffusion and flow language models spanning ~110M to ~3B parameters show that SHS consistently improves sample quality across numbers of function evaluations (NFE). Code is available at this https URL.
Accepted at NeurIPS 2026. 31 pages, 4 figures, 9 tables
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2601.02799 [cs.LG]
(or arXiv:2601.02799v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2601.02799 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 315 cuvinte
Subiecte conexe
The Design Space of Tri-Modal Masked Diffusion Models
Apple · 14 oct. 2026
FluxLite: Inference-Time Proposal Control for Discrete Diffusion Models
arXiv · 30 sept. 2026
How can LLM RL Work Despite Information-Theoretic Inefficiency
LessWrong · 16 oct. 2026
Specificity-Aware Diffusion Steering via Variance-Reduced Sequential Monte Carlo
arXiv · 2 oct. 2026