aiqbench analysis
This paper introduces E-MoE, a novel architecture for masked diffusion models that utilizes expert-routing decisions to capture positional correlations without increasing active parameters. By addressing the limitations of factorized reverse processes in few-step regimes, the method improves sample quality on benchmarks like LM1B and binarized MNIST while avoiding common issues like posterior collapse.
E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models
arXiv:2609.37533v1 Announce Type: new Abstract: Masked diffusion models (MDMs) generate sequences by progressively unmasking several tokens per denoising step, but their reverse process is typically factorized over positions, limiting sample quality in the few-step regime where diffusion's speed advantage over autoregressive decoding matters most. A recent line of work introduces a continuous Gaussian latent, trained as a variational autoencoder, to capture correlations across positions, but such approaches are prone to posterior collapse, where the latent is silently ignored. We propose Enhanced Mixture-of-Experts (E-MoE), which builds the reverse process as a mixture of factorized distributions over a discrete shared latent given by the expert-routing decisions of a Mixture-of-Experts (MoE) backbone, without increasing active parameters over the factorized baseline. Across synthetic multi-modal benchmarks, binarized MNIST, and LM1B, E-MoE improves few-step generation over factorized baselines.
Abstract:Masked diffusion models (MDMs) generate sequences by progressively unmasking several tokens per denoising step, but their reverse process is typically factorized over positions, limiting sample quality in the few-step regime where diffusion's speed advantage over autoregressive decoding matters most. A recent line of work introduces a continuous Gaussian latent, trained as a variational autoencoder, to capture correlations across positions, but such approaches are prone to posterior collapse, where the latent is silently ignored. We propose Enhanced Mixture-of-Experts (E-MoE), which builds the reverse process as a mixture of factorized distributions over a discrete shared latent given by the expert-routing decisions of a Mixture-of-Experts (MoE) backbone, without increasing active parameters over the factorized baseline. Across synthetic multi-modal benchmarks, binarized MNIST, and LM1B, E-MoE improves few-step generation over factorized baselines.
Computation and Language (cs.CL)
arXiv:2609.37533 [cs.CL]
(or arXiv:2609.37533v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.37533 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 222 words
Related coverage
ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models
arXiv · Oct 2, 2026
NVIDIA and Local AI Community Fuel Open Source Models and Intelligent Agents
NVIDIA · Aug 11, 2026
NVIDIA boosts BioNeMo models with efficient MoE training
Quantum Zeitgeist · Sep 25, 2026
Opaque recurrence, and other AI terms that you should probably know
TechCrunch · Sep 7, 2026