aiqbench analysis
This study provides an essential, rigorous benchmark of eight attention mechanism variants by explicitly measuring their total energy consumption during training on GPT-2 architectures. The findings demonstrate that optimized kernel implementations like Flash Attention significantly improve efficiency, while also clarifying that minimizing instantaneous GPU power is insufficient without considering the trade-off against increased training duration. This work offers critical practical guidance for researchers and engineers aiming to select resource-efficient mechanisms in large-scale model development.
Attentions Under the Microscope: A Comparative Study of Resource Utilization for Variants of Self-Attention
arXiv:2507.07247v2 Announce Type: replace-cross Abstract: As large language models (LLMs) and visual language models (VLMs) grow in scale and application, attention mechanisms have become a central computational bottleneck due to their high memory and time complexity. While many efficient attention variants have been proposed, there remains a lack of rigorous evaluation on their actual energy usage and hardware resource demands during training. In this work, we benchmark eight attention mechanisms in training GPT-2 architecture, measuring key metrics including training time, GPU memory usage, FLOPS, CPU usage, and power consumption. Our results reveal that attention mechanisms with optimized kernel implementations, including Flash Attention, Locality-Sensitive Hashing (LSH) Attention, and Multi-Head Latent Attention (MLA), achieve the best energy efficiency. We further show that lower GPU power alone does not guarantee reduced energy use, as training time plays an equally important role. Our study highlights the importance of energy-aware benchmarking in attention design and provides a practical insight for selecting resource-efficient mechanisms. All our codes are available at GitHub.
Abstract:As large language models (LLMs) and visual language models (VLMs) grow in scale and application, attention mechanisms have become a central computational bottleneck due to their high memory and time complexity. While many efficient attention variants have been proposed, there remains a lack of rigorous evaluation on their actual energy usage and hardware resource demands during training. In this work, we benchmark eight attention mechanisms in training GPT-2 architecture, measuring key metrics including training time, GPU memory usage, FLOPS, CPU usage, and power consumption. Our results reveal that attention mechanisms with optimized kernel implementations, including Flash Attention, Locality-Sensitive Hashing (LSH) Attention, and Multi-Head Latent Attention (MLA), achieve the best energy efficiency. We further show that lower GPU power alone does not guarantee reduced energy use, as training time plays an equally important role. Our study highlights the importance of energy-aware benchmarking in attention design and provides a practical insight for selecting resource-efficient mechanisms. All our codes are available at GitHub.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE)
arXiv:2507.07247 [cs.LG]
(or arXiv:2507.07247v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2507.07247 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 260 words
Related coverage
[AINews] not much happened today
Latent Space · Oct 3, 2026
ITP-STDP: A Hardware-Efficient Intrinsic-Timing Power-of-Two Synaptic Learning Engine for On-Chip SNNs
arXiv · Sep 1, 2026
ITP-STDP: An Intrinsic-Timing Power-of-Two Learning Engine for On-Chip SNN Training
arXiv · Jun 6, 2026
Receding-Horizon Next-Best-View Planner for Autonomous Leaf Surface Reconstruction
arXiv · Aug 3, 2026