aiqbench analysis
This research introduces Low-Rank Attention Residuals (LR-AttnRes), a novel architecture for Large Language Models that reduces computational costs by using low-dimensional keys for routing while maintaining full-dimensional values. Empirical results on 1B and 4B parameter models demonstrate that this approach lowers validation loss and increases training throughput compared to standard attention mechanisms. The authors have released the corresponding code, kernels, and trained models to support further investigation into efficient model scaling.
Low-Rank Attention Residuals
arXiv:2607.09694v2 Announce Type: replace Abstract: Attention Residuals (AttnRes) replace the fixed residual sum with depth-wise attention over previous sub-layer outputs in Large Language Models (LLMs), but use each output as both a full-dimensional key and value. This couples routing with representation and makes the cost of computing depth-routing scores scale with hidden width $d$. We propose Low-Rank Attention Residuals (LR-AttnRes), which keep full-dimensional residual values while using $r$-dimensional keys, with $r < d$, for routing. LR-AttnRes uses the last $r$ dimensions of each value as the routing key, reducing total residual-side FLOPs while still improving performance. Comprehensive sweeps across the number of blocks ($N$) and $r$ show that depth-wise routing can be effective with far fewer dimensions than the model width. At both $1$B and $4$B parameters with $r = d/4$, LR-AttnRes achieves lower final validation loss, higher average downstream accuracy, and higher measured training-step throughput than standard AttnRes. We also provide a fused kernel supporting standard and low-rank routing. We release all code, the kernel, and all trained models to facilitate future research.
Abstract:Attention Residuals (AttnRes) replace the fixed residual sum with depth-wise attention over previous sub-layer outputs in Large Language Models (LLMs), but use each output as both a full-dimensional key and value. This couples routing with representation and makes the cost of computing depth-routing scores scale with hidden width $d$. We propose Low-Rank Attention Residuals (LR-AttnRes), which keep full-dimensional residual values while using $r$-dimensional keys, with $r < d$, for routing. LR-AttnRes uses the last $r$ dimensions of each value as the routing key, reducing total residual-side FLOPs while still improving performance. Comprehensive sweeps across the number of blocks ($N$) and $r$ show that depth-wise routing can be effective with far fewer dimensions than the model width. At both $1$B and $4$B parameters with $r = d/4$, LR-AttnRes achieves lower final validation loss, higher average downstream accuracy, and higher measured training-step throughput than standard AttnRes. We also provide a fused kernel supporting standard and low-rank routing. We release all code, the kernel, and all trained models to facilitate future research.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2607.09694 [cs.LG]
(or arXiv:2607.09694v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.09694 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 265 words
Related coverage
Enabling DeepSeek-V4-Flash Training on AMD Instinct MI355X GPUs with Primus
AMD · Sep 3, 2026
Reflections on unlearning and inoculation
LessWrong · Sep 20, 2026
Spectral-Guided Diffusion: Accelerating Inference via Static Spectral Layer Scheduling
arXiv · Sep 26, 2026
Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention
arXiv · Jul 20, 2026