analiza aiqbench
This paper introduces Tucker bottleneck attention (TuBA), a novel architecture that leverages low-rank tensor decomposition to overcome the quadratic complexity limits of standard self-attention in multi-dimensional sequence modeling. By projecting hidden tensors into compact cores for efficient global token mixing, TuBA achieves significant reductions in computational cost and error rates for tasks like video prediction and weather forecasting without sacrificing accuracy. The method demonstrates a viable path toward scaling transformer-based models to longer sequences and higher dimensionalities while maintaining subquadratic computation.
Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling
arXiv:2610.09090v1 Announce Type: new Abstract: The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tensor structure for efficient global token mixing. TuBA projects hidden tensors into compact Tucker cores, performs multi-head self-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores. On video prediction and global weather forecasting, TuBA achieves favorable accuracy-efficiency trade-offs over standard and efficient attention and task-specific models. Compared to standard self-attention, TuBA reduces error and computation by up to 24.7% and 66.6% for video prediction and 37.1% and 85.1% for autoregressive weather forecasting, with speedups up to 4.27 times. Low-rank Tucker cores and multi-frame generation also outperform full-rank attention and frame-by-frame generation, respectively.
Abstract:The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tensor structure for efficient global token mixing. TuBA projects hidden tensors into compact Tucker cores, performs multi-head self-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores. On video prediction and global weather forecasting, TuBA achieves favorable accuracy-efficiency trade-offs over standard and efficient attention and task-specific models. Compared to standard self-attention, TuBA reduces error and computation by up to 24.7% and 66.6% for video prediction and 37.1% and 85.1% for autoregressive weather forecasting, with speedups up to 4.27 times. Low-rank Tucker cores and multi-frame generation also outperform full-rank attention and frame-by-frame generation, respectively.
Machine Learning (cs.LG)
arXiv:2610.09090 [cs.LG]
(or arXiv:2610.09090v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.09090 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 230 cuvinte
Subiecte conexe
Enabling DeepSeek-V4-Flash Training on AMD Instinct MI355X GPUs with Primus
AMD · 3 sept. 2026
Productionizing TurboQuant on AMD GPUs for KV-Cache-Bound LLM Inference
AMD · 11 iun. 2026
Some Intuitions on Steering Vectors
LessWrong · 29 sept. 2026
PagedAttention vs. RadixAttention: Optimizing LLM KV Cache Management
Analytics Vidhya · 21 aug. 2026