aiqbench analysis
TeDiServe introduces a cluster-level serving system specifically designed to handle the unique inference characteristics of diffusion language models, addressing challenges like confidence-based denoising and approximate KV caching. By implementing deadline-aware scheduling and dynamic cluster reconfiguration, the system significantly improves SLO attainment and reduces latency while maintaining model accuracy. This work is critical for deploying next-generation generative AI models that rely on parallel token generation in production environments.
TeDiServe: High SLO Attainment Serving for Diffusion Language Models
arXiv:2606.29094v2 Announce Type: replace Abstract: Diffusion language models (DLMs) have recently emerged as a promising alternative to conventional autoregressive language models. By generating multiple tokens in parallel during each denoising step, they offer higher inference throughput while maintaining competitive quality. However, realizing these throughput gains while meeting latency SLOs in a serving system requires addressing challenges introduced by DLMs' unique characteristics. These include navigating the speed-quality tradeoff created by confidence-based denoising, choosing appropriate parallelization levels across model instances under fluctuating load, and coordinating approximate KV caching mechanisms that introduce non-uniform per-step costs. To address these challenges, we present TeDiServe, a cluster-level serving system for DLMs. TeDiServe enables deadline-aware scheduling and adaptive load control through confidence-threshold adjustment, and dynamically reconfigures the cluster by solving a quality-aware optimization problem, while explicitly modeling the step-level heterogeneity introduced by approximate KV caching. Across multiple benchmarks and real-world traces, TeDiServe improves SLO attainment by up to 56.6 percentage points and reduces end-to-end request latency by up to 46\% while incurring less than 1\% accuracy drop.
Abstract:Diffusion language models (DLMs) have recently emerged as a promising alternative to conventional autoregressive language models. By generating multiple tokens in parallel during each denoising step, they offer higher inference throughput while maintaining competitive quality. However, realizing these throughput gains while meeting latency SLOs in a serving system requires addressing challenges introduced by DLMs' unique characteristics. These include navigating the speed-quality tradeoff created by confidence-based denoising, choosing appropriate parallelization levels across model instances under fluctuating load, and coordinating approximate KV caching mechanisms that introduce non-uniform per-step costs. To address these challenges, we present TeDiServe, a cluster-level serving system for DLMs. TeDiServe enables deadline-aware scheduling and adaptive load control through confidence-threshold adjustment, and dynamically reconfigures the cluster by solving a quality-aware optimization problem, while explicitly modeling the step-level heterogeneity introduced by approximate KV caching. Across multiple benchmarks and real-world traces, TeDiServe improves SLO attainment by up to 56.6 percentage points and reduces end-to-end request latency by up to 46\% while incurring less than 1\% accuracy drop.
Machine Learning (cs.LG)
arXiv:2606.29094 [cs.LG]
(or arXiv:2606.29094v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.29094 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 257 words
Related coverage
Efficient Vision-Language-Action Management and Serving for Robot Factories
arXiv · Sep 14, 2026
ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priors
arXiv · Oct 1, 2026
Evaluating task vectors, unlearning and inoculation
LessWrong · Sep 20, 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026