analiza aiqbench
This paper introduces FRAC, a novel State Space Model architecture that substitutes standard ODE-based dynamics with fractional calculus to achieve power-law memory retention. By approximating heavy-tailed kernels with finite-state exponential modes, the method enables efficient parallel training and prefill while significantly outperforming existing SSM baselines on long-context language modeling tasks. The work represents a substantive architectural advance in machine learning for handling extended temporal sequences without sacrificing computational efficiency.
Fractional State Space Transition for Long Sequence Modeling
arXiv:2609.36314v1 Announce Type: cross Abstract: State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Most modern SSMs rely on ODE-based dynamics that lead to exponential forgetting, limiting their ability to retain information over broad temporal ranges. We introduce FRAC, a selective SSM architecture derived from fractional dynamics that replaces this exponential decay with power-law long memory. To make fractional dynamics practical, FRAC approximates the heavy-tailed target kernel with a finite-state, log-spaced sum of exponential modes. This construction turns fractional memory into an efficient recurrent module with parallel training and prefill, while retaining bounded-state autoregressive decoding. Extensive experiments, including 1.3B-parameter language modeling, demonstrate that FRAC consistently improves long-context performance over state-of-the-art SSM baselines while staying competitive on short-context. These results show that fractional dynamics provide a practical and effective prior for long-context SSMs.
Abstract:State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Most modern SSMs rely on ODE-based dynamics that lead to exponential forgetting, limiting their ability to retain information over broad temporal ranges. We introduce FRAC, a selective SSM architecture derived from fractional dynamics that replaces this exponential decay with power-law long memory. To make fractional dynamics practical, FRAC approximates the heavy-tailed target kernel with a finite-state, log-spaced sum of exponential modes. This construction turns fractional memory into an efficient recurrent module with parallel training and prefill, while retaining bounded-state autoregressive decoding. Extensive experiments, including 1.3B-parameter language modeling, demonstrate that FRAC consistently improves long-context performance over state-of-the-art SSM baselines while staying competitive on short-context. These results show that fractional dynamics provide a practical and effective prior for long-context SSMs.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2609.36314 [cs.LG]
(or arXiv:2609.36314v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.36314 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 240 cuvinte
Subiecte conexe
Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs
LessWrong · 27 iul. 2026
Can we change an AI model’s architecture without retraining it from scratch?
Amazon Science · 12 oct. 2026
Angles of attack for continual learning safety
LessWrong · 16 iun. 2026
Deconfusing Corrigibility
AI Alignment Forum · 17 oct. 2026