analiza aiqbench
This paper formalizes insertion language models (ILMs) using a continuous-time Markov chain framework, demonstrating that existing ad-hoc methods are special cases of this unified denoising objective. Empirical results on synthetic planning tasks confirm the approach retains ILM advantages over left-to-right and masked diffusion models while providing greater sampling flexibility. The work establishes a rigorous theoretical foundation for variable-length sequence generation, offering a competitive alternative to standard autoregressive architectures.
A Continuous-Time Markov Chain Framework for Insertion Language Models
arXiv:2606.10199v1 Announce Type: new Abstract: Insertion Language Models (ILMs) offer several advantages over left-to-right generation and mask-based generation. However, existing formulations of insertion-based generation have largely been ad-hoc. In this paper, we derive a diffusion-style denoising objective for ILMs from first principles by formulating the noising process as a continuous-time Markov chain on the space of variable-length sequences. We show that previous formulations of ILMs can be viewed as special cases of this denoising framework. Through empirical evaluation on a synthetic planning task, we show that the proposed approach retains the benefits of insertion-based generation over left-to-right generation and masked diffusion models. In language modeling, our diffusion-based approach is competitive with left-to-right generation and masked diffusion models, while offering additional flexibility in sampling compared to existing insertion language models.
Abstract:Insertion Language Models (ILMs) offer several advantages over left-to-right generation and mask-based generation. However, existing formulations of insertion-based generation have largely been ad-hoc. In this paper, we derive a diffusion-style denoising objective for ILMs from first principles by formulating the noising process as a continuous-time Markov chain on the space of variable-length sequences. We show that previous formulations of ILMs can be viewed as special cases of this denoising framework. Through empirical evaluation on a synthetic planning task, we show that the proposed approach retains the benefits of insertion-based generation over left-to-right generation and masked diffusion models. In language modeling, our diffusion-based approach is competitive with left-to-right generation and masked diffusion models, while offering additional flexibility in sampling compared to existing insertion language models.
Accepted at AISTATS 2026. Code is available at this https URL
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2606.10199 [cs.LG]
(or arXiv:2606.10199v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.10199 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 233 cuvinte