aiqbench analysis
OpWeave introduces a novel framework for operator-level disaggregation in LLM serving, enabling flexible partitioning of inference stages across heterogeneous GPU clusters. By jointly optimizing operator boundaries and deployment configurations, the system achieves significant cost reductions while maintaining strict latency service objectives. This advancement addresses critical efficiency challenges in scaling large language models on diverse hardware architectures.
OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving
arXiv:2609.14237v1 Announce Type: cross Abstract: LLM serving systems increasingly disaggregate inference into finer-grained stages, with recent approaches separating attention from FFN or MoE execution during decode. This operator-level disaggregated serving (ODS) can improve hardware matching and enable independent scaling, particularly across heterogeneous devices. However, existing systems fix operator boundaries and lack a unified characterization of when disaggregation reduces serving cost. We present OpWeave, an end-to-end framework for heterogeneous ODS. OpWeave provides an analytical cost model that bounds the gains of homogeneous and heterogeneous ODS over colocated serving. It jointly optimizes operator partitioning and deployment configuration through a regularity-aware planner that keeps the search tractable even for hybrid-attention models. A vLLM-based runtime executes the synthesized plans with flexible operator stages across heterogeneous device groups. In our evaluation, OpWeave reduces serving cost by up to $1.78\times$ on homogeneous and $1.89\times$ on heterogeneous GPU clusters relative to the best feasible baseline, while meeting latency SLOs.
Abstract:LLM serving systems increasingly disaggregate inference into finer-grained stages, with recent approaches separating attention from FFN or MoE execution during decode. This operator-level disaggregated serving (ODS) can improve hardware matching and enable independent scaling, particularly across heterogeneous devices. However, existing systems fix operator boundaries and lack a unified characterization of when disaggregation reduces serving cost. We present OpWeave, an end-to-end framework for heterogeneous ODS. OpWeave provides an analytical cost model that bounds the gains of homogeneous and heterogeneous ODS over colocated serving. It jointly optimizes operator partitioning and deployment configuration through a regularity-aware planner that keeps the search tractable even for hybrid-attention models. A vLLM-based runtime executes the synthesized plans with flexible operator stages across heterogeneous device groups. In our evaluation, OpWeave reduces serving cost by up to $1.78\times$ on homogeneous and $1.89\times$ on heterogeneous GPU clusters relative to the best feasible baseline, while meeting latency SLOs.
Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI)
arXiv:2609.14237 [cs.DC]
(or arXiv:2609.14237v1 [cs.DC] for this version)
https://doi.org/10.48550/arXiv.2609.14237 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 247 words
Related coverage
ATOMesh: Unlocking AMD Hardware for Scalable LLM Serving
AMD · Jun 16, 2026
Amazon SageMaker Inference: 2026 year-to-date launches in review
AWS · Sep 18, 2026
NVIDIA’s EPD for multimodal AI
Quantum Zeitgeist · Sep 10, 2026
NVIDIA NVLink: The Scale-Up Network for AI Factories
NVIDIA · Jul 20, 2026