analiza aiqbench
This research tackles the bottleneck where NVIDIA Blackwell's FP4 tensor cores fail to accelerate attention mechanisms due to softmax conversion overhead and quantization dependencies. The authors propose "Direct-P" for noncausal inference and a novel causal path that integrates forward quantization directly into backward passes, achieving significant throughput gains on GB200 hardware. Crucially, the study identifies training divergence risks with standard MXFP4 probability/value trajectories, offering specific architectural adjustments to maintain stability in large-scale model updates.
Hardware-Aware FP4 FlashAttention-4
arXiv:2609.04105v1 Announce Type: new Abstract: Blackwell's 4-bit floating-point (FP4) tensor cores do not automatically make attention faster because softmax conversion and on-chip dependencies dominate once its matrix products shrink. We address this with \emph{Direct-P} for noncausal inference and a causal path that passes the forward quantization directly into backward. Direct-P maps scores directly to FP4 probabilities and reaches up to 2.13$\times$ the bfloat16 (BF16) forward throughput on an NVIDIA GB200. The causal path reconstructs probabilities from saved quantized queries and keys and uses 8-bit floating-point (FP8) gradient operands, accelerating a complete single-GPU 8-billion-parameter update by up to 1.14$\times$. Matched distributed training retains FP8 probabilities and values; every tested MXFP4 probability/value training trajectory diverges.
Abstract:Blackwell's 4-bit floating-point (FP4) tensor cores do not automatically make attention faster because softmax conversion and on-chip dependencies dominate once its matrix products shrink. We address this with \emph{Direct-P} for noncausal inference and a causal path that passes the forward quantization directly into backward. Direct-P maps scores directly to FP4 probabilities and reaches up to 2.13$\times$ the bfloat16 (BF16) forward throughput on an NVIDIA GB200. The causal path reconstructs probabilities from saved quantized queries and keys and uses 8-bit floating-point (FP8) gradient operands, accelerating a complete single-GPU 8-billion-parameter update by up to 1.14$\times$. Matched distributed training retains FP8 probabilities and values; every tested MXFP4 probability/value training trajectory diverges.
Machine Learning (cs.LG)
arXiv:2609.04105 [cs.LG]
(or arXiv:2609.04105v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.04105 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 202 cuvinte
Subiecte conexe
GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model
Meta Engineering · 3 aug. 2026
From Naive to Near-Peak: Building High-Performance GEMM Kernels with Gluon
AMD · 22 mai 2026
7 Approaches to Efficient LLM Training on Limited Hardware
KDnuggets · 9 sept. 2026
Building Blocks for Foundation Model Training and Inference on AWS
Hugging Face · 11 mai 2026