analiza aiqbench
EntropyInfer introduces a training-free framework that dynamically allocates compute to individual attention heads based on real-time entropy patterns, addressing the limitations of fixed sparsity in long-context inference. By distinguishing between rigid and dynamic head behaviors, the method achieves up to 2.39× speedup beyond 100k tokens with minimal quality loss across major model families. This approach represents a significant step forward for efficient LLM deployment, particularly for applications requiring extended context windows without retraining.
From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs
arXiv:2606.09508v1 Announce Type: cross Abstract: Existing sparse attention and KV cache compression methods for long-context LLM inference typically apply fixed sparsity patterns or uniform budgets across all attention heads, overlooking the substantial variation in attention behavior among heads and contexts. We observe two distinct entropy patterns among attention heads: Rigid Heads, whose entropy stays near zero across input segments, and Dynamic Heads, whose entropy fluctuates significantly. Crucially, the distribution of these types is context-dependent and cannot be predetermined offline. We therefore propose EntropyInfer, a training-free framework that uses attention entropy to adaptively allocate compute at the granularity of individual heads and segments during prefilling. For decoding, we introduce a latent KV cache compression scheme that leverages generated output tokens, rather than prefill tokens alone, to identify and retain the most critical cache entries. Extensive experiments on Llama, Qwen and openPangu model series show that EntropyInfer consistently outperforms baselines including SnapKV, AdaKV, and CritiPrefill, achieving up to 2.39$\times$ end-to-end speedup beyond 100k tokens with minimal quality degradation compared to full attention. The code is released in https://github.com/SHA-4096/EntropyInfer.
Abstract:Existing sparse attention and KV cache compression methods for long-context LLM inference typically apply fixed sparsity patterns or uniform budgets across all attention heads, overlooking the substantial variation in attention behavior among heads and contexts. We observe two distinct entropy patterns among attention heads: Rigid Heads, whose entropy stays near zero across input segments, and Dynamic Heads, whose entropy fluctuates significantly. Crucially, the distribution of these types is context-dependent and cannot be predetermined offline. We therefore propose EntropyInfer, a training-free framework that uses attention entropy to adaptively allocate compute at the granularity of individual heads and segments during prefilling. For decoding, we introduce a latent KV cache compression scheme that leverages generated output tokens, rather than prefill tokens alone, to identify and retain the most critical cache entries. Extensive experiments on Llama, Qwen and openPangu model series show that EntropyInfer consistently outperforms baselines including SnapKV, AdaKV, and CritiPrefill, achieving up to 2.39$\times$ end-to-end speedup beyond 100k tokens with minimal quality degradation compared to full attention. The code is released in this https URL.
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2606.09508 [cs.AI]
(or arXiv:2606.09508v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.09508 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 271 cuvinte
Subiecte conexe
AdaMEM: Test-Time Adaptive Memory for Language Agents
arXiv · 6 iun. 2026
Adaptive Oscillatory-State Alignment for Time Series Forecasting
arXiv · 15 iun. 2026
AGMark: Attention-Guided Dynamic Watermarking for Large Vision-Language Models
arXiv · 29 iul. 2026
Developing Healthcare Robotics with GPU-Native Medical Physics Simulation
NVIDIA · 28 iul. 2026