aiqbench analysis
Block Sparse Flash Attention (BSFA) offers a training-free optimization for long-context inference by dynamically pruning non-essential attention blocks based on exact similarity scores. This approach reduces computational load and memory transfers by roughly 50% while maintaining high accuracy, effectively addressing the quadratic complexity bottleneck in modern large language models. The method serves as a direct drop-in replacement for existing FlashAttention implementations, delivering significant speedups across various model architectures and GPU hardware.
Block Sparse Flash Attention
arXiv:2512.07011v2 Announce Type: replace-cross Abstract: Modern large language models increasingly require long contexts for reasoning and multi-document tasks, but attention's quadratic complexity creates a severe computational bottleneck. We present Block Sparse Flash Attention (BSFA), a drop-in replacement that accelerates long-context inference while preserving model quality. Unlike methods that predict importance before computing scores, BSFA computes exact query-key similarities to select the top-k most important value blocks for each query. By comparing per-block maximum scores against calibrated thresholds, we skip approximately 50% of the computation and memory transfers for pruned blocks. Our training-free approach requires only a one-time threshold calibration on a small dataset to learn the per-layer and per-head attention score distributions. We provide a CUDA kernel implementation that can be used as a drop-in replacement for FlashAttention. On Llama-3.1-8B, BSFA achieves up to 1.13x end-to-end speedup on LongBench with only a 1.1% accuracy drop, and up to 1.24x on Needle-in-a-Haystack retrieval at a 1% accuracy drop. The attention kernel itself accelerates by up to 1.38x. We compare BSFA against five recent sparse attention baselines (SpargeAttention, MInference, FlexPrefill, XAttention, and BLASST), and verify the method on Qwen2.5-7B and on A6000 and H100 GPUs. The implementation is available at https://github.com/Danielohayon/Block-Sparse-Flash-Attention.
Abstract:Modern large language models increasingly require long contexts for reasoning and multi-document tasks, but attention's quadratic complexity creates a severe computational bottleneck. We present Block Sparse Flash Attention (BSFA), a drop-in replacement that accelerates long-context inference while preserving model quality. Unlike methods that predict importance before computing scores, BSFA computes exact query-key similarities to select the top-k most important value blocks for each query. By comparing per-block maximum scores against calibrated thresholds, we skip approximately 50% of the computation and memory transfers for pruned blocks. Our training-free approach requires only a one-time threshold calibration on a small dataset to learn the per-layer and per-head attention score distributions. We provide a CUDA kernel implementation that can be used as a drop-in replacement for FlashAttention. On Llama-3.1-8B, BSFA achieves up to 1.13x end-to-end speedup on LongBench with only a 1.1% accuracy drop, and up to 1.24x on Needle-in-a-Haystack retrieval at a 1% accuracy drop. The attention kernel itself accelerates by up to 1.38x. We compare BSFA against five recent sparse attention baselines (SpargeAttention, MInference, FlexPrefill, XAttention, and BLASST), and verify the method on Qwen2.5-7B and on A6000 and H100 GPUs. The implementation is available at this https URL.
Accepted to NeurIPS 2026. 16 pages, 3 figures, 7 tables. Code: this https URL
Machine Learning (cs.LG); Computation and Language (cs.CL); Performance (cs.PF)
arXiv:2512.07011 [cs.LG]
(or arXiv:2512.07011v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2512.07011 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 308 words
Related coverage
Enabling DeepSeek-V4-Flash Training on AMD Instinct MI355X GPUs with Primus
AMD · Sep 3, 2026
GLM-5.3-Flash vs Qwen3.8-Flash-Next: Two Chinese AI Labs Independently Converge on the Same Model Architecture
MarkTechPost · Aug 28, 2026
VSA: Accelerating Video Diffusion Inference with Sparse Attention on AMD GPUs
AMD · Aug 4, 2026
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026