analiza aiqbench
EvoSpec addresses critical bottlenecks in speculative decoding by dynamically adapting vocabulary and draft model parameters using lightweight LoRA updates. This approach achieves significant inference speedups while reducing memory overhead compared to full-parameter online adaptation methods, directly enhancing the efficiency of large language model deployment.
EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation
arXiv:2605.27390v3 Announce Type: replace Abstract: Speculative decoding accelerates Large Language Model inference through draft-then-verify generation, yet lightweight draft models face coupled efficiency and quality limitations: large-vocabulary output projection is costly, while limited draft capacity and static parameters reduce acceptance under specialized or shifting inputs. Vocabulary pruning lowers projection cost, but static variants miss locally important long-tail tokens, while dynamic variants remain sensitive to preset selection policies and budgets. Moreover, limited draft capacity can leave the draft distribution misaligned even when the target token is covered. Online alignment improves draft quality, but full-parameter updates introduce substantial memory and latency overhead. We introduce EvoSpec, which jointly adapts the active vocabulary and lightweight draft parameters from verification feedback. EvoSpec asynchronously retrieves semantic and statistical token neighbors and performs curriculum-weighted online LoRA alignment while preserving exact target-model verification. On Qwen3-8B/EAGLE-2, EvoSpec reaches a $2.18\times$ speedup over vanilla decoding and a $1.20\times$ gain over EAGLE-2, while improving specialized-domain coverage and using $27\%$ less auxiliary GPU adaptation memory than full-parameter online adaptation.
Abstract:Speculative decoding accelerates Large Language Model inference through draft-then-verify generation, yet lightweight draft models face coupled efficiency and quality limitations: large-vocabulary output projection is costly, while limited draft capacity and static parameters reduce acceptance under specialized or shifting inputs. Vocabulary pruning lowers projection cost, but static variants miss locally important long-tail tokens, while dynamic variants remain sensitive to preset selection policies and budgets. Moreover, limited draft capacity can leave the draft distribution misaligned even when the target token is covered. Online alignment improves draft quality, but full-parameter updates introduce substantial memory and latency overhead. We introduce EvoSpec, which jointly adapts the active vocabulary and lightweight draft parameters from verification feedback. EvoSpec asynchronously retrieves semantic and statistical token neighbors and performs curriculum-weighted online LoRA alignment while preserving exact target-model verification. On Qwen3-8B/EAGLE-2, EvoSpec reaches a $2.18\times$ speedup over vanilla decoding and a $1.20\times$ gain over EAGLE-2, while improving specialized-domain coverage and using $27\%$ less auxiliary GPU adaptation memory than full-parameter online adaptation.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2605.27390 [cs.CL]
(or arXiv:2605.27390v3 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 176 cuvinte
Subiecte conexe
Model Weight Profiles: Where Do the Parameters Go?
AMD · 25 sept. 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · 13 oct. 2026
[AINews] Muse Spark 1.3 matches GPT-5.6-Sol, confirming Meta Superintelligence as the newest Frontier Lab, >90% discount for training
Latent Space · 3 sept. 2026
[AINews] Agents disprove Polysemanticity; greatly improve Interpretability & Alignment
Latent Space · 16 oct. 2026