aiqbench analysis
MiniMax has introduced MSA, a two-branch sparse attention mechanism that significantly reduces compute costs for long-context inference by restricting softmax operations to selected key-value blocks rather than processing the entire sequence. This architecture maintains performance comparable to Grouped Query Attention while achieving an 28.4× reduction in per-token attention computation at scale of one million tokens. The method represents a notable advancement in optimizing efficiency for large-scale Mixture-of-Experts models without sacrificing accuracy on downstream benchmarks.
MiniMax Sparse Attention (MSA): a Two-Branch Block-Sparse Attention Trained on a 109B-Parameter MoE With a 3T-Token Budget
MiniMax released MSA, a sparse attention built on Grouped Query Attention. A lightweight Index Branch selects Top-k key-value blocks per query and GQA group; the Main Branch attends only to those blocks. It matches GQA on downstream benchmarks while reducing per-token attention compute 28.4× at 1M context. The post MiniMax Sparse Attention (MSA): a Two-Branch Block-Sparse Attention Trained on a 109B-Parameter MoE With a 3T-Token Budget appeared first on MarkTechPost.
MiniMax released MSA (MiniMax Sparse Attention), a sparse attention method built directly on Grouped Query Attention (GQA). It targets one bottleneck: the quadratic cost of softmax attention at long context. The MiniMax research team tested it inside a 109B-parameter Mixture-of-Experts model trained with native multimodal data. They also open-sourced an inference kernel and shipped a production model, MiniMax-M3. MSA (MiniMax Sparse Attention) factors attention into two stages: an Index Branch and a Main Branch. The Index Branch decides which key-value blocks each query should read. The Main Branch then runs exact softmax attention over only those blocks. Selection happens at block granularity, not per token. The default block size is…
Excerpt shown under the publisher's RSS terms · 110 words · full text and figures are on the publisher's site
Related coverage
GLM-5.3-Flash vs Qwen3.8-Flash-Next: Two Chinese AI Labs Independently Converge on the Same Model Architecture
MarkTechPost · Aug 28, 2026
MiniMax Sparse Attention
arXiv · Jun 12, 2026
MiniMax Sparse Attention
arXiv · Jun 15, 2026
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026