analiza aiqbench
MiniMax has introduced MSA, a two-branch sparse attention mechanism that significantly reduces compute costs for long-context inference by restricting softmax operations to selected key-value blocks rather than processing the entire sequence. This architecture maintains performance comparable to Grouped Query Attention while achieving an 28.4× reduction in per-token attention computation at scale of one million tokens. The method represents a notable advancement in optimizing efficiency for large-scale Mixture-of-Experts models without sacrificing accuracy on downstream benchmarks.
MiniMax Sparse Attention (MSA): a Two-Branch Block-Sparse Attention Trained on a 109B-Parameter MoE With a 3T-Token Budget
MiniMax released MSA, a sparse attention built on Grouped Query Attention. A lightweight Index Branch selects Top-k key-value blocks per query and GQA group; the Main Branch attends only to those blocks. It matches GQA on downstream benchmarks while reducing per-token attention compute 28.4× at 1M context. The post MiniMax Sparse Attention (MSA): a Two-Branch Block-Sparse Attention Trained on a 109B-Parameter MoE With a 3T-Token Budget appeared first on MarkTechPost.
MiniMax released MSA (MiniMax Sparse Attention), a sparse attention method built directly on Grouped Query Attention (GQA). It targets one bottleneck: the quadratic cost of softmax attention at long context. The MiniMax research team tested it inside a 109B-parameter Mixture-of-Experts model trained with native multimodal data. They also open-sourced an inference kernel and shipped a production model, MiniMax-M3. MSA (MiniMax Sparse Attention) factors attention into two stages: an Index Branch and a Main Branch. The Index Branch decides which key-value blocks each query should read. The Main Branch then runs exact softmax attention over only those blocks. Selection happens at block granularity, not per token. The default block size is…
Extras afișat conform termenilor RSS ai publicației · 110 cuvinte · textul integral și figurile sunt pe site-ul publicației
Subiecte conexe
GLM-5.3-Flash vs Qwen3.8-Flash-Next: Two Chinese AI Labs Independently Converge on the Same Model Architecture
MarkTechPost · 28 aug. 2026
MiniMax Sparse Attention
arXiv · 12 iun. 2026
MiniMax Sparse Attention
arXiv · 15 iun. 2026
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026