aiqbench analysis
This paper is about LLM inference serving: it uses speculative decoding, where a drafter proposes tokens that the target model verifies in one pass, and applies it to structured outputs whose fields are linked by exact relations. The reported gains (up to 10.5 tokens per verify step and up to 1.43x faster decoding in SGLang) are measured on the authors' benchmarks and serving engines, so they should be read as results for this setup rather than general claims about LLM speed. For readers, the main takeaway is that enforcing semantic constraints during decoding, not just syntax, may matter for reliable structured generation.
FactorSpec: Serving Exact Relational Constraints as Speculative Drafts
arXiv:2611.02297v1 Announce Type: new Abstract: Speculative decoding speeds up LLM serving by drafting tokens that the target model verifies in one pass, and its gain grows with the drafts the target accepts. Structured outputs should be an ideal case, because much of a record follows from relations among its fields: an itinerary's days follow from its cities' stays and flights. Serving engines, however, draft such text by proposing it from the prompt or a learned head, and their grammar masks enforce only syntax, so the model must satisfy the relations, which it violates: on NATURAL PLAN's trip problems, Qwen3-8B writes a valid itinerary for 10%, and retries that name the violated relation raise this to 14%. We present FactorSpec, which serves exact relational constraints as speculative drafts. FactorSpec compiles a JSON Schema and its relations into a factor graph with an exact feasibility check, drafts every token the constraint determines as certain, verifies each request at a width chosen for it, and computes the masks in worker processes outside the scheduler while the GPU runs. In SGLang, its drafts yield up to 10.5 tokens per verify step, up to 1.9x those of the fastest n-gram drafter, and it decodes up to 1.43x faster than speculators tuned per load, at a grammar mask's cost; it also leads in vLLM. Because every record is also valid, it serves up to 13.9x the valid records per GPU-second of syntax-only baselines.
Abstract:Speculative decoding speeds up LLM serving by drafting tokens that the target model verifies in one pass, and its gain grows with the drafts the target accepts. Structured outputs should be an ideal case, because much of a record follows from relations among its fields: an itinerary's days follow from its cities' stays and flights. Serving engines, however, draft such text by proposing it from the prompt or a learned head, and their grammar masks enforce only syntax, so the model must satisfy the relations, which it violates: on NATURAL PLAN's trip problems, Qwen3-8B writes a valid itinerary for 10%, and retries that name the violated relation raise this to 14%. We present FactorSpec, which serves exact relational constraints as speculative drafts. FactorSpec compiles a JSON Schema and its relations into a factor graph with an exact feasibility check, drafts every token the constraint determines as certain, verifies each request at a width chosen for it, and computes the masks in worker processes outside the scheduler while the GPU runs. In SGLang, its drafts yield up to 10.5 tokens per verify step, up to 1.9x those of the fastest n-gram drafter, and it decodes up to 1.43x faster than speculators tuned per load, at a grammar mask's cost; it also leads in vLLM. Because every record is also valid, it serves up to 13.9x the valid records per GPU-second of syntax-only baselines.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2611.02297 [cs.LG]
(or arXiv:2611.02297v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2611.02297 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 329 words
Related coverage
Distributed LLM Inference via Communication-Efficient Speculative Decoding
arXiv · Oct 28, 2026
S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices
arXiv · Aug 18, 2026
Justice After Identity: Large Language Models and the View from Everywhere
arXiv · Oct 8, 2026
Princeton University Team Models Speculative Window Decoding for Slower Gate Technologies
Quantum Zeitgeist · Jul 15, 2026