Don't Commit Alone: Joint Token Commitment in Diffusion Language Models

Abstract:Diffusion language models (dLLMs) commit multiple tokens per denoising step by decoding each selected position independently from a shared context. When these positions are dependent, this factorization introduces an error captured by conditional total correlation, which confidence-based selection cannot infer from marginal probabilities alone. We propose CoCommit, a marker-gated coordination pass that delays commitment. After the usual bundle selection, a learned marker identifies the commit set, and the backbone's last n layers are re-applied to coordinate the marked positions before greedy argmax writes the tokens. This approximates joint-mode decoding while reusing existing weights, requiring only one partial forward pass and no auxiliary model. On LLaDA 2.1 with LoRA adapters and greedy inference, joint commitment improves five of the seven evaluated benchmarks over the released factorized decoder. The largest gains occur on code and reasoning tasks, while the remaining tasks are near parity.

Computation and Language (cs.CL)

arXiv:2607.04469 [cs.CL]

(or arXiv:2607.04469v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2607.04469 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.