Inexact calculus of variations on the hyperspherical tangent bundle with connections to the attention mechanism

Abstract:We offer a theoretical mathematical background through Lagrangian optimization on the unit hyperspherical manifold and its tangential structure. Our methods can be categorized as inexact since our methods are projection-based and since we will perturb the functional optimization with epsilon-type quantities. We draw connections to the attention mechanism and the Transformer since it exists as a flow map in the tangent fiber for each token along the high-dimensional unit sphere. Our motivation for this work is primarily twofold: we study the attention mechanism under its flow map and its relations to traditional calculus of variations and Lagrangian optimization; and we study several calculus problems of variations on the unit hypersphere that appeal to a broader mathematical lens in approximating, variational contexts.

Machine Learning (cs.LG)

arXiv:2507.15431 [cs.LG]

(or arXiv:2507.15431v5 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2507.15431 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.