aiqbench analysis
This paper is about LLM serving systems: PACE, a per-step admission controller that decides how many parallel decoding branches to run in a batch so that throughput gains don't degrade co-batched requests. Its reported results (1.48x to 1.77x goodput gains on Qwen3-32B at 99% SLO attainment, and similar gains across six models, the largest up to 671B parameters) are benchmark claims from the authors and have not been independently verified. For AI infrastructure engineers, the practical takeaway is that the "safe" parallelism level for inference is dynamic and depends on batch state, so static concurrency caps and eager branching are both likely suboptimal.
Pacing Branch Parallelism in LLM Serving
arXiv:2605.06914v2 Announce Type: replace-cross Abstract: Recent methods expose intra-request parallelism in LLM outputs, allowing independent branches to decode concurrently. Existing serving systems execute these branches eagerly or under fixed caps. We show that both are brittle: eager admission inflates the shared decode step, degrading co-batched requests in serial stages, while conservative fixed caps forgo the throughput that motivated exposing branches in the first place. We call the excess step latency caused by admitted branches the \emph{branch externality} and show that the safe width depends on batch composition, context lengths, and accumulated slack, all of which change continuously over a workload trace. We introduce PACE, a per-step admission controller that treats extra branches as opportunistic width, admitted only when the predicted branch externality fits within the batch's current slack budget. Per-step regulation is practical because branch-level scheduling decouples compute from memory: branches share the request's prefix KV, so expanding or contracting width requires no memory reclamation. On Qwen3-32B, PACE improves goodput by $1.77\times$ over serving without branch parallelism and by $1.48\times$ over eager admission while maintaining 99\% SLO attainment. Comparable gains ($1.72$--$2.15\times$) hold on five additional dense and mixture-of-experts models with up to 671B parameters. Code is available at https://github.com/Swapnil-Gandhi/PACE
Abstract:Recent methods expose intra-request parallelism in LLM outputs, allowing independent branches to decode concurrently. Existing serving systems execute these branches eagerly or under fixed caps. We show that both are brittle: eager admission inflates the shared decode step, degrading co-batched requests in serial stages, while conservative fixed caps forgo the throughput that motivated exposing branches in the first place. We call the excess step latency caused by admitted branches the \emph{branch externality} and show that the safe width depends on batch composition, context lengths, and accumulated slack, all of which change continuously over a workload trace. We introduce PACE, a per-step admission controller that treats extra branches as opportunistic width, admitted only when the predicted branch externality fits within the batch's current slack budget. Per-step regulation is practical because branch-level scheduling decouples compute from memory: branches share the request's prefix KV, so expanding or contracting width requires no memory reclamation. On Qwen3-32B, PACE improves goodput by $1.77\times$ over serving without branch parallelism and by $1.48\times$ over eager admission while maintaining 99\% SLO attainment. Comparable gains ($1.72$--$2.15\times$) hold on five additional dense and mixture-of-experts models with up to 671B parameters. Code is available at this https URL
Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2605.06914 [cs.DC]
(or arXiv:2605.06914v2 [cs.DC] for this version)
https://doi.org/10.48550/arXiv.2605.06914 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 298 words
Related coverage
Accelerate LLM model loading and increase context windows with GPUDirect on Amazon FSx for Lustre and TurboQuant
AWS · Jun 1, 2026
SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
arXiv · Sep 30, 2026
FastEEP: Fast Intra-Instance Elastic Scaling for Expert-Parallel MoE Serving
arXiv · Oct 14, 2026
FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications
arXiv · Jul 21, 2026