aiqbench analysis
PrefillShare addresses a critical bottleneck in multi-agent systems by enabling shared prefill computation and KV caches across multiple fine-tuned models within disaggregated serving architectures. By factorizing models into static prefill and dynamic decode modules, this approach eliminates redundant processing of common prompt prefixes, significantly reducing tail latency and increasing throughput without sacrificing accuracy. The technique represents a meaningful advancement in optimizing inference efficiency for complex, multi-model workflows where shared context is standard.
PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving
arXiv:2602.12029v2 Announce Type: replace Abstract: Multi-agent systems increasingly orchestrate multiple specialized language models to solve complex real-world problems, often invoking them over a shared context. This execution pattern repeatedly processes the same prompt prefix across models. Consequently, each model redundantly executes the prefill stage and maintains its own key-value (KV) cache, increasing aggregate prefill load and worsening tail latency by intensifying prefill-decode interference in existing LLM serving stacks. Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs, but disaggregation does not fundamentally eliminate inter-model redundancy in computation and KV storage for the same prompt. To address this issue, we propose PrefillShare, a novel algorithm that enables sharing the prefill stage across multiple fine-tuned models in a disaggregated setting. PrefillShare factorizes the model into prefill and decode modules, freezes the prefill module, and fine-tunes only the decode module. This design allows multiple task-specific models to share a prefill module and the KV cache generated for the same prompt. We further introduce a routing mechanism that enables effective prefill sharing in a vLLM-based disaggregated system. PrefillShare not only matches full fine-tuning accuracy on a broad range of tasks and models, but also delivers 4.5x lower p95 latency and 2.0x higher throughput in multi-model agent workloads.
Abstract:Multi-agent systems increasingly orchestrate multiple specialized language models to solve complex real-world problems, often invoking them over a shared context. This execution pattern repeatedly processes the same prompt prefix across models. Consequently, each model redundantly executes the prefill stage and maintains its own key-value (KV) cache, increasing aggregate prefill load and worsening tail latency by intensifying prefill-decode interference in existing LLM serving stacks. Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs, but disaggregation does not fundamentally eliminate inter-model redundancy in computation and KV storage for the same prompt. To address this issue, we propose PrefillShare, a novel algorithm that enables sharing the prefill stage across multiple fine-tuned models in a disaggregated setting. PrefillShare factorizes the model into prefill and decode modules, freezes the prefill module, and fine-tunes only the decode module. This design allows multiple task-specific models to share a prefill module and the KV cache generated for the same prompt. We further introduce a routing mechanism that enables effective prefill sharing in a vLLM-based disaggregated system. PrefillShare not only matches full fine-tuning accuracy on a broad range of tasks and models, but also delivers 4.5x lower p95 latency and 2.0x higher throughput in multi-model agent workloads.
Preprint. 22 pages, 8 figures
Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)
arXiv:2602.12029 [cs.LG]
(or arXiv:2602.12029v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.12029 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 305 words
Related coverage
DI Series: Serving Kimi-K3 MXFP4 Wide-EP Disaggregated on AMD Instinct MI300X and MI325X
AMD · Oct 8, 2026
DI Series: Scaling GLM-5.1-FP8 to 64 MI300X GPUs
AMD · Aug 21, 2026
[AINews] not much happened today
Latent Space · Sep 10, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026