analiza aiqbench
DiffuMamba addresses the quadratic complexity bottleneck of Transformer-based diffusion language models by integrating a bidirectional Mamba backbone for linear-time sequence modeling. This architecture achieves up to 8.2x higher inference throughput on long sequences while maintaining performance parity with standard autoregressive and transformer-diffusion baselines across scales reaching 1.3B parameters. The findings suggest that cache-efficient block diffusion combined with state-space mixers is the optimal path for scaling future generative systems without sacrificing speed or quality.
DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone
arXiv:2511.15927v4 Announce Type: replace-cross Abstract: Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead. We introduce DiffuMamba, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling, and DiffuMamba-H, a hybrid variant with interleaved attention. Across scales up to 1.3B parameters, our models match Transformer-based diffusion in downstream performance while achieving up to 8.2x and 4.3x higher inference throughput, respectively, on long sequences. We further present a systematic analysis of inference efficiency across modern DLM variants combining asymptotic complexity with empirical measurements. Notably, cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly with sequence length and achieves the strongest performance across all baselines, suggesting a promising direction for future diffusion-based generation systems.
Abstract:Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead. We introduce DiffuMamba, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling, and DiffuMamba-H, a hybrid variant with interleaved attention. Across scales up to 1.3B parameters, our models match Transformer-based diffusion in downstream performance while achieving up to 8.2x and 4.3x higher inference throughput, respectively, on long sequences. We further present a systematic analysis of inference efficiency across modern DLM variants combining asymptotic complexity with empirical measurements. Notably, cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly with sequence length and achieves the strongest performance across all baselines, suggesting a promising direction for future diffusion-based generation systems.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2511.15927 [cs.LG]
(or arXiv:2511.15927v4 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 157 cuvinte
Subiecte conexe
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026
Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
AWS · 4 sept. 2026
Google AI Releases DiffusionGemma, a 26B MoE Open Model Using Text Diffusion for Up to 4x Faster Generation
MarkTechPost · 10 iun. 2026
ManiFusion: Unlocking the High-Throughput Generation via Superposition in Manifold Space
arXiv · 16 oct. 2026