analiza aiqbench
This paper proposes Dynamical Kernel Attention (DKA), a novel mechanism that integrates stochastic dynamics into self-attention to model how information decays over time rather than relying solely on static similarity metrics. By theoretically bounding the evidence derived from past tokens, DKA aims to mitigate recency biases while maintaining the computational efficiency and parallel structure of standard attention mechanisms. The authors note that empirical validation is deferred to a future release, focusing this work primarily on the theoretical framework.
Dynamical Kernel Attention: Estimating the Present from a Record of the Past
arXiv:2610.24026v1 Announce Type: new Abstract: Standard self-attention compares tokens by similarity but does not account for how much of a past state survives to the present. We introduce Dynamical Kernel Attention (DKA), which derives attention from robust estimation of a latent state evolving under stochastic dynamics. As a past state decays toward equilibrium under the dynamics, the information it carries about the present vanishes, bounding the total evidence a head can draw. Unlike recency biases, DKA can abstain when relevant evidence is weak and guarantees a bounded error when old tokens are discarded. DKA preserves the parallel structure and computational cost of standard attention. We present the model and its theory; empirical evaluation is deferred to a subsequent version.
Abstract:Standard self-attention compares tokens by similarity but does not account for how much of a past state survives to the present. We introduce Dynamical Kernel Attention (DKA), which derives attention from robust estimation of a latent state evolving under stochastic dynamics. As a past state decays toward equilibrium under the dynamics, the information it carries about the present vanishes, bounding the total evidence a head can draw. Unlike recency biases, DKA can abstain when relevant evidence is weak and guarantees a bounded error when old tokens are discarded. DKA preserves the parallel structure and computational cost of standard attention. We present the model and its theory; empirical evaluation is deferred to a subsequent version.
Machine Learning (cs.LG)
arXiv:2610.24026 [cs.LG]
(or arXiv:2610.24026v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.24026 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 206 cuvinte
Subiecte conexe
Human-Oriented World Model
arXiv · 4 nov. 2026
Deep Bayesian Approaches for Interpretable Nonlinear Structural System Identification
arXiv · 28 oct. 2026
A Quantum Optimization Framework for Data-Assimilation-Augmented Parameter Estimation
arXiv · 17 aug. 2026
Reinforcement Learning with Inner-loop Dynamics Estimator for Aerial Manipulation under Uncertainty
arXiv · 16 iun. 2026