aiqbench analysis
WhiFlash addresses the inference bottleneck in large language models by dynamically switching between autoregressive and diffusion-based drafting strategies at the token level. This cross-paradigm approach utilizes fine-grained routing policies and optimized cache management to significantly increase throughput, particularly for complex agentic workloads where static methods underperform. The technique effectively balances latency and generation speed by leveraging the complementary strengths of distinct architectural paradigms.
WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing
arXiv:2606.07710v2 Announce Type: replace Abstract: The autoregressive nature of large language models (LLMs) remains a significant bottleneck for inference, particularly in complex agentic workloads. While speculative decoding (SD) accelerates inference, current approaches rely on static drafting paradigms, utilising either autoregressive drafting models for reasoning or diffusion-based parallel drafting models for structured outputs. We empirically find that drafting accuracy fluctuates dramatically within a single sequence, leaving significant performance unrealised by static paradigms and coarse-grained routing. To address this volatility, we introduce WhiFlash, the first cross-paradigm SD method that unifies autoregressive and diffusion-based parallel drafting under a single token-level controller. WhiFlash adopts a fine-grained routing mechanism that employs either a lightweight entropy-based or a learned neural policy, both parametrised to provide a tunable balance between expected token gain and latency. To make high-frequency switching computationally viable, we introduce novel cache-management optimisations, Lazy Catch-up and KV-only Prefill, reducing switching overhead to below 7% of per-round latency. By capitalising on the complementary strengths of fundamentally distinct drafting architectures, WhiFlash achieves significantly higher acceptance lengths, yielding category-specific throughput gains of up to 69.6% over the state-of-the-art autoregressive EAGLE-3 and 37.3% over the diffusion-based DFlash.
Abstract:The autoregressive nature of large language models (LLMs) remains a significant bottleneck for inference, particularly in complex agentic workloads. While speculative decoding (SD) accelerates inference, current approaches rely on static drafting paradigms, utilising either autoregressive drafting models for reasoning or diffusion-based parallel drafting models for structured outputs. We empirically find that drafting accuracy fluctuates dramatically within a single sequence, leaving significant performance unrealised by static paradigms and coarse-grained routing. To address this volatility, we introduce WhiFlash, the first cross-paradigm SD method that unifies autoregressive and diffusion-based parallel drafting under a single token-level controller. WhiFlash adopts a fine-grained routing mechanism that employs either a lightweight entropy-based or a learned neural policy, both parametrised to provide a tunable balance between expected token gain and latency. To make high-frequency switching computationally viable, we introduce novel cache-management optimisations, Lazy Catch-up and KV-only Prefill, reducing switching overhead to below 7% of per-round latency. By capitalising on the complementary strengths of fundamentally distinct drafting architectures, WhiFlash achieves significantly higher acceptance lengths, yielding category-specific throughput gains of up to 69.6% over the state-of-the-art autoregressive EAGLE-3 and 37.3% over the diffusion-based DFlash.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2606.07710 [cs.LG]
(or arXiv:2606.07710v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.07710 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 279 words
Related coverage
WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing
arXiv · Jun 9, 2026
Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference
NVIDIA · Sep 2, 2026
Efficient MiniMax-M3 Inference on AMD Instinct GPUs with ATOM and ATOMesh
AMD · Jul 21, 2026
BV Loss: Block Verification-Aware Loss for Block Diffusion Speculative Decoding
arXiv · Sep 30, 2026