analiza aiqbench
This study investigates how post-training transforms autoregressive models into masked diffusion variants, revealing a task-dependent shift in computational mechanisms rather than a uniform replacement. The findings indicate that while prefix-dominant tasks preserve inherited pathways, globally constrained tasks trigger significant reorganization toward earlier layers with more diffuse component specialization. These insights clarify the architectural dynamics of non-autoregressive generation, offering critical data for researchers optimizing inference efficiency and model structure.
Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models
arXiv:2601.14758v5 Announce Type: replace-cross Abstract: Post-training pretrained autoregressive models (ARMs) into masked diffusion models (MDMs) provides an efficient route to diffusion language modeling, but it remains unclear whether the resulting models reuse inherited autoregressive computation or reorganize it for non-autoregressive generation. We compare two 7B ARM-MDM families across four controlled diagnostic tasks and find a task-dependent mechanism shift. On prefix-dominant tasks, MDMs largely preserve inherited high-attribution pathways or exhibit only modest changes in where computation occurs. On globally constrained tasks, the reorganization is substantially stronger, with task-relevant computation shifting toward earlier layers. This depth-wise pattern persists across prompt resampling, circuit budgets, and tested inference budgets, while targeted ablations support the functional importance of the identified structures under the tested intervention protocols. At the component level, diagnostic probes suggest that ARMs rely more strongly on sharply specialized components, whereas MDMs exhibit weaker single-component specialization and more diffuse output-space alignment. Together, these results suggest that diffusion post-training selectively preserves or reorganizes inherited computation according to task structure, rather than uniformly replacing autoregressive mechanisms.
Abstract:Post-training pretrained autoregressive models (ARMs) into masked diffusion models (MDMs) provides an efficient route to diffusion language modeling, but it remains unclear whether the resulting models reuse inherited autoregressive computation or reorganize it for non-autoregressive generation. We compare two 7B ARM-MDM families across four controlled diagnostic tasks and find a task-dependent mechanism shift. On prefix-dominant tasks, MDMs largely preserve inherited high-attribution pathways or exhibit only modest changes in where computation occurs. On globally constrained tasks, the reorganization is substantially stronger, with task-relevant computation shifting toward earlier layers. This depth-wise pattern persists across prompt resampling, circuit budgets, and tested inference budgets, while targeted ablations support the functional importance of the identified structures under the tested intervention protocols. At the component level, diagnostic probes suggest that ARMs rely more strongly on sharply specialized components, whereas MDMs exhibit weaker single-component specialization and more diffuse output-space alignment. Together, these results suggest that diffusion post-training selectively preserves or reorganizes inherited computation according to task structure, rather than uniformly replacing autoregressive mechanisms.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2601.14758 [cs.LG]
(or arXiv:2601.14758v5 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2601.14758 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 265 cuvinte
Subiecte conexe
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · 25 sept. 2026
Humanity’s Last Invention — Richard Socher of Recursive
Latent Space · 14 sept. 2026
Parallelize speculative decoding with P-EAGLE on Amazon SageMaker AI
AWS · 16 iun. 2026
Rethinking the Divergence Regularization in LLM RL
arXiv · 9 iun. 2026