analiza aiqbench
FreeSolo introduces a novel sparse attention architecture that achieves linear-time complexity while enabling zero-shot extrapolation beyond training contexts, addressing key scalability bottlenecks in long-sequence modeling. By combining short-range sliding windows with a long-range keypoint selection mechanism, it offers a promising alternative to hybrid architectures that incur quadratic costs. This work advances the efficiency and generalization capabilities of transformer-based models for handling extended context lengths without expensive fine-tuning.
FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation
arXiv:2610.17191v1 Announce Type: cross Abstract: Linear-time sequence models typically store long-context information in a recurrent state. A fixed-size state limits how much information can be stored, while hybrid architectures that add full attention for better recall incur quadratic sequence cost. Alternatively, sparse attention with continued pretraining (CPT) can extend the effective context length. Generalization beyond the trained length remains challenging, however, and CPT can be costly. We introduce FreeSolo, a native sparse attention mechanism with a short-range sliding window branch and a long-range branch that selects keypoint cells for direct token attention. Its linear-time variant, FS-L, learns to prune the prefix before query-dependent selection through short CPT. FS-B searches balanced trees with $O(N\log^2 N)$ algorithmic cost including radix-sort construction. FreeSolo demonstrates better zero-shot length extrapolation than the baselines without long-context fine-tuning.
Abstract:Linear-time sequence models typically store long-context information in a recurrent state. A fixed-size state limits how much information can be stored, while hybrid architectures that add full attention for better recall incur quadratic sequence cost. Alternatively, sparse attention with continued pretraining (CPT) can extend the effective context length. Generalization beyond the trained length remains challenging, however, and CPT can be costly. We introduce FreeSolo, a native sparse attention mechanism with a short-range sliding window branch and a long-range branch that selects keypoint cells for direct token attention. Its linear-time variant, FS-L, learns to prune the prefix before query-dependent selection through short CPT. FS-B searches balanced trees with $O(N\log^2 N)$ algorithmic cost including radix-sort construction. FreeSolo demonstrates better zero-shot length extrapolation than the baselines without long-context fine-tuning.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.17191 [cs.LG]
(or arXiv:2610.17191v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.17191 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 223 cuvinte
Subiecte conexe
Synthesis Superintelligence: from Semiconductors to Superconductors — Periodic Labs’ Liam Fedus and Ekin Dogus Cubuk
Latent Space · 8 oct. 2026
[AINews] Meta Connect 2026: Muse glasses, voice, video, and Charm
Latent Space · 24 sept. 2026
Links #5: 2026/07
LessWrong · 31 iul. 2026
[AINews] The Future of Latent Space
Latent Space · 25 sept. 2026