analiza aiqbench
This paper introduces JPEG-DLM, a novel approach that jointly trains compressors and diffusion models to learn structured, efficient compressed embeddings for language generation. By optimizing the embedding space during training rather than fixing it beforehand, the method significantly improves throughput and reduces perplexity on large-scale benchmarks like LM1B and OWT. This work represents a meaningful advancement in making continuous diffusion language models more computationally viable without sacrificing generation quality.
One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusion
arXiv:2609.33698v1 Announce Type: new Abstract: Most continuous diffusion language models process one latent position per token at each sampling step, making generation expensive. Two-stage methods lower the cost by reducing the latent length, but they fix the compressed embedding space before training the diffusion model. Embeddings from the fixed space can be difficult to model with diffusion and decode reliably into tokens, which limits generation quality after compression. To address this problem, we introduce JPEG-DLM (Joint-embedding Prediction for Efficient Generation with Diffusion Language Model), which jointly trains a compressor, a flow matching model and a decoding module. With joint-embedding prediction, JPEG-DLM learns compressed embeddings that are more structured, easier to model with diffusion and reliably decodable into tokens. JPEG-DLM achieves the lowest mean Gen-PPL and highest throughput among recent diffusion and flow models on LM1B and OWT. At a compression rate of 0.5 on OWT, it reaches a Gen-PPL of 34.52 and approximately 2.3 times ELF's throughput. These results suggest that jointly learning compressed embeddings offers a promising path toward efficient diffusion language modeling. Code will be released soon.
Abstract:Most continuous diffusion language models process one latent position per token at each sampling step, making generation expensive. Two-stage methods lower the cost by reducing the latent length, but they fix the compressed embedding space before training the diffusion model. Embeddings from the fixed space can be difficult to model with diffusion and decode reliably into tokens, which limits generation quality after compression. To address this problem, we introduce JPEG-DLM (Joint-embedding Prediction for Efficient Generation with Diffusion Language Model), which jointly trains a compressor, a flow matching model and a decoding module. With joint-embedding prediction, JPEG-DLM learns compressed embeddings that are more structured, easier to model with diffusion and reliably decodable into tokens. JPEG-DLM achieves the lowest mean Gen-PPL and highest throughput among recent diffusion and flow models on LM1B and OWT. At a compression rate of 0.5 on OWT, it reaches a Gen-PPL of 34.52 and approximately 2.3 times ELF's throughput. These results suggest that jointly learning compressed embeddings offers a promising path toward efficient diffusion language modeling. Code will be released soon.
25 pages, 7 figures, and 8 tables. Includes appendices
Artificial Intelligence (cs.AI)
arXiv:2609.33698 [cs.AI]
(or arXiv:2609.33698v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.33698 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 277 cuvinte
Subiecte conexe
World Models and Yann LeCun’s Bet Against Large Language Models
Quantum Zeitgeist · 17 iul. 2026
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · 23 sept. 2026
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · 17 sept. 2026
Structural Process Supervision for Latent Chain-of-Thought Reasoning
arXiv · 11 sept. 2026