analiza aiqbench
NAVIRA introduces a decoupled inference policy for masked diffusion language models that separates quality scoring from token regeneration to mitigate error propagation. By stochastically remasking unreliable tokens in a second forward pass, the method improves fluency and LLM-judge scores while preserving entropy without relying solely on learned quality signals. This approach highlights that refining the remasking strategy is as critical as the underlying quality signal for reliable text generation in diffusion architectures.
NAVIRA: Decoupled Stochastic Remasking for Masked Diffusion Language Models
arXiv:2606.06031v1 Announce Type: new Abstract: Masked diffusion language models generate text by iteratively unmasking many tokens in parallel, but this speed comes with a correction problem: tokens generated in the same step are predicted from marginal distributions, and early local dependency errors can later contaminate the context. PRISM addresses this by learning token-level quality scores and remasking unreliable tokens, but its inference rule is coupled: the same forward pass both detects low-quality tokens and computes logits for their replacements, so the erroneous tokens still condition regeneration. We propose NAVIRA, an inference-time decoding policy that separates these two operations and samples remasking positions stochastically. A first forward pass scores tokens; selected tokens are masked; a second forward pass regenerates from the cleaned context. Temperature-controlled remasking reduces repeated correction of the same positions and balances fluency against diversity. In controlled experiments with a 170M masked diffusion language model, decoupling improves fluency, while scheduled stochastic remasking preserves entropy and achieves stronger LLM-judge scores under larger forward-pass budgets. These results show that remasking policy, not only the learned quality signal, is central to reliable masked-diffusion text generation.
Abstract:Masked diffusion language models generate text by iteratively unmasking many tokens in parallel, but this speed comes with a correction problem: tokens generated in the same step are predicted from marginal distributions, and early local dependency errors can later contaminate the context. PRISM addresses this by learning token-level quality scores and remasking unreliable tokens, but its inference rule is coupled: the same forward pass both detects low-quality tokens and computes logits for their replacements, so the erroneous tokens still condition regeneration. We propose NAVIRA, an inference-time decoding policy that separates these two operations and samples remasking positions stochastically. A first forward pass scores tokens; selected tokens are masked; a second forward pass regenerates from the cleaned context. Temperature-controlled remasking reduces repeated correction of the same positions and balances fluency against diversity. In controlled experiments with a 170M masked diffusion language model, decoupling improves fluency, while scheduled stochastic remasking preserves entropy and achieves stronger LLM-judge scores under larger forward-pass budgets. These results show that remasking policy, not only the learned quality signal, is central to reliable masked-diffusion text generation.
Computation and Language (cs.CL)
arXiv:2606.06031 [cs.CL]
(or arXiv:2606.06031v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.06031 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 273 cuvinte
Subiecte conexe
CaRE Compute-aware Remasking Evaluation Protocol for Masked Diffusion Language Models
arXiv · 29 iul. 2026
Re-evaluating Confidence Remasking in Masked Diffusion Language Models
arXiv · 12 iun. 2026
Diffusion Language Models as Open Quantum Systems
arXiv · 16 oct. 2026
The Design Space of Tri-Modal Masked Diffusion Models
Apple · 14 oct. 2026