analiza aiqbench
PreDiff-LM introduces a hybrid attention mechanism that reconciles causal pretraining with bidirectional denoising to improve discrete masked diffusion language models. While the method significantly reduces training steps and enhances generation quality over uniform baselines, it still lags behind compute-matched fine-tuned autoregressive models in raw performance metrics like perplexity. This work clarifies how attention adaptation can serve as a complementary strategy for adapting pretrained backbones without claiming full weight reuse novelty.
PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention
arXiv:2607.25157v1 Announce Type: new Abstract: Discrete masked diffusion language models support bidirectional generation and infilling, but adapting pretrained autoregressive (AR) transformers requires reconciling causal pretraining with bidirectional denoising. We study this problem at the level of attention rather than claiming AR-weight reuse itself as novel. PreDiff-LM preserves causal attention within the observed prompt while allowing full bidirectional attention within the masked target. Under a matched GPT-2 Medium, WikiText-103, 90K-step setup, this hybrid mask improves unconditional perplexity from 34.1 to 28.7 and MAUVE from 0.71 to 0.78 over uniform bidirectional attention with the same AR initialization. Attention adaptation also composes with a DiffuGPT-style objective adaptation, reaching 26.9 perplexity. Pretrained initialization reduces the steps required to reach perplexity below 50 from about 350K to 8K, although a compute-matched fine-tuned AR model remains stronger at equal scale (18.9 versus 28.7). Beyond perplexity, PreDiff-LM improves repetition, distributional quality, four zero-shot downstream tasks, and human preference over prior diffusion baselines. The results position hybrid attention as a complementary mechanism for adapting pretrained causal backbones, while making explicit the remaining quality and inference-efficiency gaps to optimized AR models.
Abstract:Discrete masked diffusion language models support bidirectional generation and infilling, but adapting pretrained autoregressive (AR) transformers requires reconciling causal pretraining with bidirectional denoising. We study this problem at the level of attention rather than claiming AR-weight reuse itself as novel. PreDiff-LM preserves causal attention within the observed prompt while allowing full bidirectional attention within the masked target. Under a matched GPT-2 Medium, WikiText-103, 90K-step setup, this hybrid mask improves unconditional perplexity from 34.1 to 28.7 and MAUVE from 0.71 to 0.78 over uniform bidirectional attention with the same AR initialization. Attention adaptation also composes with a DiffuGPT-style objective adaptation, reaching 26.9 perplexity. Pretrained initialization reduces the steps required to reach perplexity below 50 from about 350K to 8K, although a compute-matched fine-tuned AR model remains stronger at equal scale (18.9 versus 28.7). Beyond perplexity, PreDiff-LM improves repetition, distributional quality, four zero-shot downstream tasks, and human preference over prior diffusion baselines. The results position hybrid attention as a complementary mechanism for adapting pretrained causal backbones, while making explicit the remaining quality and inference-efficiency gaps to optimized AR models.
Artificial Intelligence (cs.AI)
arXiv:2607.25157 [cs.AI]
(or arXiv:2607.25157v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2607.25157 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 195 cuvinte
Subiecte conexe
SUAVE: Unified Video-Action Models via Masked Diffusion
arXiv · 6 oct. 2026
Multi-Mask Diffusion Language Models for Few-Step Generation
arXiv · 23 iul. 2026
NeoMME: an efficient Multimodal-native and Multilingual Encoder
Hugging Face · 3 sept. 2026
The Design Space of Tri-Modal Masked Diffusion Models
Apple · 14 oct. 2026