analiza aiqbench
SparseEngine addresses the memory bottlenecks of long-context LLM agents by introducing a sparse-first inference architecture that unifies heterogeneous cache representations. This approach enables cross-request state management and selective KV-cache pruning, delivering significant throughput gains over standard engines like vLLM without sacrificing model quality. The system demonstrates that optimizing for sparse attention workflows is critical for scaling agent-based applications with extensive interaction histories.
SparseEngine: Sparse-First Inference Engine
arXiv:2609.39068v1 Announce Type: new Abstract: Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these costs, heterogeneous cache representations and workflows hinder integration with existing inference engines, while prior sparse-serving abstractions support only specific layouts or workflows. We present SparseEngine, a ground-up, sparse-first inference engine whose shared lifecycle contract lets each method control its KV representation and computation while coordinating state transitions with common serving infrastructure. SparseEngine supports 15 methods across four categories and enables cross-request state management through Chain Cache, which resumes KV-eviction methods from retained history, and controllable Prefix-Cache Pruning, which removes KV from selected history regions while preserving logical-prefix matching. While maintaining method quality, SparseEngine delivers over 10x higher throughput with KV eviction, over 2.5x faster decoding at matched concurrency than vLLM, and over 2x end-to-end speedup on agent benchmarks. The code is available at https://github.com/CURRENTF/SparseEngine.
Abstract:Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these costs, heterogeneous cache representations and workflows hinder integration with existing inference engines, while prior sparse-serving abstractions support only specific layouts or workflows. We present SparseEngine, a ground-up, sparse-first inference engine whose shared lifecycle contract lets each method control its KV representation and computation while coordinating state transitions with common serving infrastructure. SparseEngine supports 15 methods across four categories and enables cross-request state management through Chain Cache, which resumes KV-eviction methods from retained history, and controllable Prefix-Cache Pruning, which removes KV from selected history regions while preserving logical-prefix matching. While maintaining method quality, SparseEngine delivers over 10x higher throughput with KV eviction, over 2.5x faster decoding at matched concurrency than vLLM, and over 2x end-to-end speedup on agent benchmarks. The code is available at this https URL.
Machine Learning (cs.LG)
arXiv:2609.39068 [cs.LG]
(or arXiv:2609.39068v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.39068 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 238 cuvinte
Subiecte conexe
Efficient MiniMax-M3 Inference on AMD Instinct GPUs with ATOM and ATOMesh
AMD · 21 iul. 2026
Serving DeepSeek-V4: why million-token context is an inference systems problem
Together AI · 11 mai 2026
[AINews] not much happened today
Latent Space · 3 oct. 2026
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026