aiqbench analysis
TopK-Guided introduces a training-free optimization technique that dynamically allocates sparsity budgets across transformer blocks to improve LLM inference efficiency without sacrificing accuracy. By addressing the limitations of existing threshold-based and fixed-sparsity methods, this approach offers a practical path for deploying larger models with reduced computational overhead. The method demonstrates consistent gains in perplexity and downstream tasks while maintaining comparable projection compute costs.
TopK-Guided: Adaptive, Budget-Aware Activation Sparsity for Efficient LLM Inference
arXiv:2610.01763v1 Announce Type: new Abstract: Activation sparsity speeds up large language model (LLM) inference by setting unimportant activations to zero so that the corresponding computations can be skipped. Existing training-free methods, however, make different trade-offs: threshold-based methods such as TEAL adapt the sparsity level to each token but do not tightly control the realised sparsity, while TopK-based methods such as WINA enforce a fixed sparsity level but use the same sparsity budget for every token. Both also apply the same budget across transformer blocks, despite large differences in block sensitivity. We introduce TopK-Guided, a training-free method that addresses both limitations by combining bounded token-level sparsity adaptation with sensitivity-aware block-level budget allocation. Across Llama-2 and Llama-3 models, TopK-Guided consistently improves perplexity and downstream accuracy over TEAL and WINA while preserving essentially the same sparsitydependent projection compute as WINA, with the largest gains at high sparsity. Ablations show that both components provide complementary improvements.
Abstract:Activation sparsity speeds up large language model (LLM) inference by setting unimportant activations to zero so that the corresponding computations can be skipped. Existing training-free methods, however, make different trade-offs: threshold-based methods such as TEAL adapt the sparsity level to each token but do not tightly control the realised sparsity, while TopK-based methods such as WINA enforce a fixed sparsity level but use the same sparsity budget for every token. Both also apply the same budget across transformer blocks, despite large differences in block sensitivity. We introduce TopK-Guided, a training-free method that addresses both limitations by combining bounded token-level sparsity adaptation with sensitivity-aware block-level budget allocation. Across Llama-2 and Llama-3 models, TopK-Guided consistently improves perplexity and downstream accuracy over TEAL and WINA while preserving essentially the same sparsitydependent projection compute as WINA, with the largest gains at high sparsity. Ablations show that both components provide complementary improvements.
Artificial Intelligence (cs.AI)
arXiv:2610.01763 [cs.AI]
(or arXiv:2610.01763v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.01763 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 241 words
Related coverage
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
Digital States 2026: AI Moves From Experiment to Infrastructure
govtech.com · Sep 24, 2026
Quantization and Pruning Methods to Make Your LLM Leaner
KDnuggets · Aug 28, 2026
[AINews] not much happened today
Latent Space · Sep 10, 2026