aiqbench analysis
Scorpio addresses a critical bottleneck in LLM serving by dynamically scheduling requests with heterogeneous Service Level Objectives, rather than treating all traffic uniformly to maximize raw throughput. By implementing specialized guards for Time to First Token and Time Per Output Token alongside predictive admission control, the system significantly improves goodput and SLO adherence under high-load conditions. This research advances the practical deployment of LLMs in production environments where diverse latency requirements must be met simultaneously.
Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference
arXiv:2505.23022v2 Announce Type: replace Abstract: Large Language Model (LLM) serving increasingly underpins online Web services such as conversational agents, Web search, and programming assistants, where requests carry heterogeneous Service Level Objectives (SLOs) such as Time to First Token (TTFT) and Time Per Output Token (TPOT). Existing LLM serving systems prioritize maximum throughput and treat all requests uniformly, which leads to suboptimal SLO attainment. This paper introduces Scorpio, an SLO-oriented LLM serving system designed to maximize system goodput and SLO attainment for workloads with heterogeneous SLOs. Our core insight is to exploit SLO heterogeneity for adaptive scheduling across admission control, queue management, and batch selection. Scorpio features a TTFT Guard, which employs least-deadline-first reordering and rejects unattainable requests, and a TPOT Guard, which utilizes a VBS-based admission control and a novel credit-based batching mechanism. Both guards are supported by a predictive module. Evaluations demonstrate that Scorpio improves system goodput by up to 14.4x and SLO adherence by up to 46.5% under high load compared to state-of-the-art baselines.
Abstract:Large Language Model (LLM) serving increasingly underpins online Web services such as conversational agents, Web search, and programming assistants, where requests carry heterogeneous Service Level Objectives (SLOs) such as Time to First Token (TTFT) and Time Per Output Token (TPOT). Existing LLM serving systems prioritize maximum throughput and treat all requests uniformly, which leads to suboptimal SLO attainment. This paper introduces Scorpio, an SLO-oriented LLM serving system designed to maximize system goodput and SLO attainment for workloads with heterogeneous SLOs. Our core insight is to exploit SLO heterogeneity for adaptive scheduling across admission control, queue management, and batch selection. Scorpio features a TTFT Guard, which employs least-deadline-first reordering and rejects unattainable requests, and a TPOT Guard, which utilizes a VBS-based admission control and a novel credit-based batching mechanism. Both guards are supported by a predictive module. Evaluations demonstrate that Scorpio improves system goodput by up to 14.4x and SLO adherence by up to 46.5% under high load compared to state-of-the-art baselines.
Accepted at WISE 2026 (27th International Conference on Web Information Systems Engineering), Special Track on Innovative Web Technologies. 15 pages. This version has been accepted for publication after peer review but is not the Version of Record; the Version of Record will appear in the Springer LNCS proceedings
Machine Learning (cs.LG)
arXiv:2505.23022 [cs.LG]
(or arXiv:2505.23022v2 [cs.LG] for this version)
Full text under the publisher's open licence · 220 words
Related coverage
When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving
NVIDIA · Sep 9, 2026
AI Serving Platform That Adapts to Your Model
Databricks · Jun 10, 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
Hint-based CoT faithfulness evals still mostly work on Claude
LessWrong · Jul 30, 2026