analiza aiqbench
ADAS introduces a training-free reranking mechanism for masked diffusion language models that mitigates inference fragility by penalizing candidates with strong attention to uncertain positions. This approach maintains the base sampler's stopping logic while improving low-NFE performance by over 9 percentage points on standard benchmarks with minimal runtime overhead. The method offers a modular, soft-constraint alternative to hard graph-based compatibility checks, enhancing parallel decoding quality without retraining.
Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models
arXiv:2606.10829v1 Announce Type: new Abstract: Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration, but this parallelism is fragile: positions that are individually confident may be unsafe to commit together when their predictions are coupled. Existing training-free samplers such as Top-\(k\), Fast-dLLM, and EB-Sampler mainly control how many tokens to reveal, while often ranking candidates by token-wise scores that ignore interactions within the selected set. We propose ADAS, a training-free reranking rule for parallel masked diffusion decoding. ADAS leaves the base sampler's stopping rule unchanged and modifies only subset construction: it greedily discounts a candidate when it attends strongly to already selected positions whose predictions remain uncertain. Unlike graph-constrained methods that turn attention into hard compatibility constraints, ADAS keeps attention continuous and uses it as a soft marginal penalty. Across LLaDA-8B-Base and Dream-7B-Base on GSM8K, MATH500, HumanEval, and MBPP, plugging ADAS into Top-\(k\), Fast-dLLM, and EB-Sampler improves low-NFE performance at matched denoiser evaluations by \(9.11\) and \(10.46\) percentage points on average, respectively, with \(3.1\%\) per-forward runtime overhead. These results show that soft attention-discounted reranking is a simple and modular way to improve quality in highly parallel decoding for masked diffusion language models.
Abstract:Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration, but this parallelism is fragile: positions that are individually confident may be unsafe to commit together when their predictions are coupled. Existing training-free samplers such as Top-\(k\), Fast-dLLM, and EB-Sampler mainly control how many tokens to reveal, while often ranking candidates by token-wise scores that ignore interactions within the selected set. We propose ADAS, a training-free reranking rule for parallel masked diffusion decoding. ADAS leaves the base sampler's stopping rule unchanged and modifies only subset construction: it greedily discounts a candidate when it attends strongly to already selected positions whose predictions remain uncertain. Unlike graph-constrained methods that turn attention into hard compatibility constraints, ADAS keeps attention continuous and uses it as a soft marginal penalty. Across LLaDA-8B-Base and Dream-7B-Base on GSM8K, MATH500, HumanEval, and MBPP, plugging ADAS into Top-\(k\), Fast-dLLM, and EB-Sampler improves low-NFE performance at matched denoiser evaluations by \(9.11\) and \(10.46\) percentage points on average, respectively, with \(3.1\%\) per-forward runtime overhead. These results show that soft attention-discounted reranking is a simple and modular way to improve quality in highly parallel decoding for masked diffusion language models.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.10829 [cs.CL]
(or arXiv:2606.10829v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.10829 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 291 cuvinte
Subiecte conexe
Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models
arXiv · 1 sept. 2026
PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention
arXiv · 29 iul. 2026
LESS Is More: Mutual-Stability Sampling for Diffusion Language Models
arXiv · 17 iun. 2026
Adapting Diffusion Language Models for Lossless Pixel-Level Image Transmission
arXiv · 6 iun. 2026