analiza aiqbench
This paper applies modern Hopfield model formalism to derive a rigorous statistical-mechanics framework for Transformer attention mechanisms. By treating attention as an equation of motion within a free-energy landscape, the authors identify critical thresholds separating glassy states from robust inference phases under stochastic query conditions. This theoretical advancement offers a fundamental understanding of phase transitions in neural network dynamics rather than proposing new architectural components or empirical benchmarks.
On the Statistical Mechanics of Attention
arXiv:2610.17347v1 Announce Type: cross Abstract: We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.
Abstract:We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.
Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn); Statistical Mechanics (cond-mat.stat-mech); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
arXiv:2610.17347 [cs.LG]
(or arXiv:2610.17347v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.17347 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 143 cuvinte
Subiecte conexe
Reviewing failure modes in science as we prepare for AI researchers
LessWrong · 8 oct. 2026
From Architecture to Output: Structural Origins of Hallucination in Large Language Models and the Amplifying Role of Data
arXiv · 12 iun. 2026
FANTF: Fuzzy attention network based transformers for time series analysis
arXiv · 16 oct. 2026
Addressing Corpus Knowledge Poisoning Attacks on RAG Using Sparse Attention
arXiv · 27 aug. 2026