analiza aiqbench
This paper provides a rigorous geometric derivation showing that standard Transformer components like attention and normalization emerge naturally from polar coordinate state estimation rather than ad-hoc design choices. By retaining specific correction terms, the proposed Polar Transformer offers a theoretically grounded alternative with distinct dynamical properties for hyperspherical data modeling. This work deepens our fundamental understanding of why current architectures succeed while opening new avenues for architecture search based on geometric constraints.
The Transformer as a Polar State Estimator
arXiv:2605.11007v3 Announce Type: replace-cross Abstract: We show that the core components of the Transformer---attention, residual connections, and normalization---arise naturally from a single geometric state estimation problem. Modeling the latent state in polar coordinates naturally separates radial and hyperspherical dynamics, yielding a precision-weighted filtering procedure in which normalization enforces the hyperspherical constraint, attention aggregates directional evidence, and the residual connection implements an incremental state update. The standard Transformer block with rotary positional encodings is recovered by discarding the geometric correction terms of the resulting state estimator, showing that its architecture follows from the underlying estimation problem rather than from independent design choices. The proposed \textit{Polar Transformer} retains these geometric corrections.
Abstract:We show that the core components of the Transformer---attention, residual connections, and normalization---arise naturally from a single geometric state estimation problem. Modeling the latent state in polar coordinates naturally separates radial and hyperspherical dynamics, yielding a precision-weighted filtering procedure in which normalization enforces the hyperspherical constraint, attention aggregates directional evidence, and the residual connection implements an incremental state update. The standard Transformer block with rotary positional encodings is recovered by discarding the geometric correction terms of the resulting state estimator, showing that its architecture follows from the underlying estimation problem rather than from independent design choices. The proposed \textit{Polar Transformer} retains these geometric corrections.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.11007 [cs.LG]
(or arXiv:2605.11007v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.11007 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 199 cuvinte
Subiecte conexe
Running Variational Quantum Eigensolver with Qiskit Aer on AMD Instinct
AMD · 29 mai 2026
Noise-Robust Quantum State Characterization for Remote State Preparation with Deep Learning
arXiv · 18 sept. 2026
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · 18 sept. 2026
Does preservation make sense before we know how to revive?
LessWrong · 15 iun. 2026