aiqbench analysis
SPECTRA addresses the computational bottleneck in LLM inference by dynamically reconfiguring FPGA tiles to handle the shifting arithmetic intensity between memory-bound decoding and compute-bound prefill phases. This adaptive architecture sustains high utilization across speculative decoding pipelines, delivering significant speedups over fixed designs without relying on AI model training or quantum principles.
SPECTRA: Adaptive Execution of Speculative Decoding on a Runtime-Reconfigurable Tiled Architecture
arXiv:2609.24847v1 Announce Type: cross Abstract: LLM inference on edge devices is constrained by computational and memory resources, making efficient autoregressive decoding challenging. Speculative decoding alleviates this bottleneck by generating tokens with a smaller draft model and verifying multiple tokens in parallel with a batched target model pass. However, verification introduces a runtime-dependent intermediate regime between memory-bound general matrix-vector (GEMV) operations in decoding and compute-bound general matrix-matrix (GEMM) operations in prefill, as its arithmetic intensity varies with speculation length and acceptance rate. We present SPECTRA, a runtime-reconfigurable tiled architecture that sustains high utilization across the full speculative decoding pipeline. Within each tile, the compute engine switches between systolic execution for GEMMs and vector-lane execution for GEMVs. Across tiles, SPECTRA dynamically adapts computation parallelism by selecting tile count, kernel partitioning, and communication pattern. Both tile-level and system-level reconfiguration operate on a per-kernel basis, enabling efficient execution across these diverse regimes. Evaluated on a 20-tile FPGA prototype across the Pythia, SmolLM2, and GPT-2 families, SPECTRA achieves up to $2.09\times$ speedup from tile-level reconfiguration and a further $1.25\times$ gain from system-level adaptability over fixed designs.
Abstract:LLM inference on edge devices is constrained by computational and memory resources, making efficient autoregressive decoding challenging. Speculative decoding alleviates this bottleneck by generating tokens with a smaller draft model and verifying multiple tokens in parallel with a batched target model pass. However, verification introduces a runtime-dependent intermediate regime between memory-bound general matrix-vector (GEMV) operations in decoding and compute-bound general matrix-matrix (GEMM) operations in prefill, as its arithmetic intensity varies with speculation length and acceptance rate. We present SPECTRA, a runtime-reconfigurable tiled architecture that sustains high utilization across the full speculative decoding pipeline. Within each tile, the compute engine switches between systolic execution for GEMMs and vector-lane execution for GEMVs. Across tiles, SPECTRA dynamically adapts computation parallelism by selecting tile count, kernel partitioning, and communication pattern. Both tile-level and system-level reconfiguration operate on a per-kernel basis, enabling efficient execution across these diverse regimes. Evaluated on a 20-tile FPGA prototype across the Pythia, SmolLM2, and GPT-2 families, SPECTRA achieves up to $2.09\times$ speedup from tile-level reconfiguration and a further $1.25\times$ gain from system-level adaptability over fixed designs.
Hardware Architecture (cs.AR); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
arXiv:2609.24847 [cs.AR]
(or arXiv:2609.24847v1 [cs.AR] for this version)
https://doi.org/10.48550/arXiv.2609.24847 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 279 words
Related coverage
Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference
NVIDIA · Sep 2, 2026
Optimizing ATOM and vLLM-ATOM for High-Interactivity Inference
AMD · Sep 1, 2026
Enabling Speculative Speculative Decoding on MI300X
AMD · May 29, 2026
Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference
NVIDIA · Jul 31, 2026