analiza aiqbench
This research introduces ADAS, a training-free reranking mechanism for masked diffusion language models that mitigates inference fragility by discounting token confidence based on attention to already selected positions. By addressing the coupling issue where individually confident predictions may be unsafe when combined, the method improves reasoning and code generation performance across multiple benchmarks with minimal runtime overhead. The work advances the efficiency of parallel denoising iterations in large language models without requiring model retraining.
Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models
arXiv:2606.10829v3 Announce Type: replace-cross Abstract: Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration, but this parallelism is fragile: positions that are individually confident may be unsafe to commit together when their predictions are coupled. Existing training-free samplers such as Top-$k$, Fast-dLLM, and EB-Sampler mainly control how many tokens to reveal, while often ranking candidates by token-wise scores that ignore interactions within the selected set. We propose ADAS, a training-free reranking rule that leaves the base sampler's stopping rule unchanged and greedily discounts each token-wise confidence score according to its attention to already selected positions, weighted by their prediction uncertainty. Across LLaDA-8B-Base and Dream-7B-Base on the reasoning benchmarks GSM8K and MATH500 and the code benchmarks HumanEval and MBPP, plugging ADAS into all three samplers improves low-NFE performance at matched denoiser evaluations by $9.11$ and $10.46$ percentage points on average, respectively, with $3.1\%$ per-forward runtime overhead.
Abstract:Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration, but this parallelism is fragile: positions that are individually confident may be unsafe to commit together when their predictions are coupled. Existing training-free samplers such as Top-$k$, Fast-dLLM, and EB-Sampler mainly control how many tokens to reveal, while often ranking candidates by token-wise scores that ignore interactions within the selected set. We propose ADAS, a training-free reranking rule that leaves the base sampler's stopping rule unchanged and greedily discounts each token-wise confidence score according to its attention to already selected positions, weighted by their prediction uncertainty. Across LLaDA-8B-Base and Dream-7B-Base on the reasoning benchmarks GSM8K and MATH500 and the code benchmarks HumanEval and MBPP, plugging ADAS into all three samplers improves low-NFE performance at matched denoiser evaluations by $9.11$ and $10.46$ percentage points on average, respectively, with $3.1\%$ per-forward runtime overhead.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.10829 [cs.CL]
(or arXiv:2606.10829v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.10829 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 242 cuvinte
Subiecte conexe
Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models
arXiv · 10 iun. 2026
PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention
arXiv · 29 iul. 2026
LESS Is More: Mutual-Stability Sampling for Diffusion Language Models
arXiv · 17 iun. 2026
Adapting Diffusion Language Models for Lossless Pixel-Level Image Transmission
arXiv · 6 iun. 2026