aiqbench analysis
This paper addresses the training-decoding discrepancy in speculative decoding by reframing draft training as variational inference over latent paths rather than single greedy trajectories. By optimizing for the marginal probability of target-model acceptance, the proposed method achieves up to a 9.6% speedup over existing baselines like EAGLE-3. The approach offers a theoretically grounded improvement in decoding efficiency that is particularly relevant for optimizing inference pipelines in large and multimodal language models.
Variational Speculative Decoding: Rethinking Draft Training from Token Likelihood to Sequence Acceptance
arXiv:2602.05774v4 Announce Type: replace Abstract: Speculative decoding accelerates inference for (M)LLMs, yet a training-decoding discrepancy persists: while existing methods optimize single greedy trajectories, decoding involves verifying and ranking multiple sampled draft paths. We propose Variational Speculative Decoding (VSD), formulating draft training as variational inference over latent proposals (draft paths). VSD maximizes the marginal probability of target-model acceptance, yielding an ELBO that promotes high-quality latent proposals while minimizing divergence from the target distribution. To enhance quality and reduce variance, we incorporate a path-level utility and optimize via an Expectation-Maximization procedure. The E-step draws Monte Carlo samples from an oracle-filtered posterior, while the M-step maximizes weighted likelihood using Adaptive Rejection Weighting (ARW) and Confidence-Aware Regularization (CAR). Theoretical analysis confirms that VSD increases expected acceptance length and speedup. Extensive experiments across LLMs and MLLMs show that VSD achieves up to a 9.6% speedup over EAGLE-3 and 7.9% over ViSpec, significantly improving decoding efficiency.
Abstract:Speculative decoding accelerates inference for (M)LLMs, yet a training-decoding discrepancy persists: while existing methods optimize single greedy trajectories, decoding involves verifying and ranking multiple sampled draft paths. We propose Variational Speculative Decoding (VSD), formulating draft training as variational inference over latent proposals (draft paths). VSD maximizes the marginal probability of target-model acceptance, yielding an ELBO that promotes high-quality latent proposals while minimizing divergence from the target distribution. To enhance quality and reduce variance, we incorporate a path-level utility and optimize via an Expectation-Maximization procedure. The E-step draws Monte Carlo samples from an oracle-filtered posterior, while the M-step maximizes weighted likelihood using Adaptive Rejection Weighting (ARW) and Confidence-Aware Regularization (CAR). Theoretical analysis confirms that VSD increases expected acceptance length and speedup. Extensive experiments across LLMs and MLLMs show that VSD achieves up to a 9.6% speedup over EAGLE-3 and 7.9% over ViSpec, significantly improving decoding efficiency.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Probability (math.PR)
arXiv:2602.05774 [cs.LG]
(or arXiv:2602.05774v4 [cs.LG] for this version)
Full text under the publisher's open licence · 161 words
Related coverage
Variational Speculative Decoding: Rethinking Draft Training from Token Likelihood to Sequence Acceptance
arXiv · Jul 24, 2026
Acceptance-Aware Draft Model Training for Speculative Decoding
arXiv · Sep 22, 2026
AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decoding
arXiv · Jun 17, 2026
AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decoding
arXiv · Jun 5, 2026