Toward Manifest Relationality in Transformers via Symmetry Reduction

Abstract:Transformer models contain substantial internal redundancy arising from coordinate-dependent representations and continuous symmetries, in model space and in head space, respectively. While recent approaches address this by explicitly breaking symmetry, we propose a complementary framework based on symmetry reduction. We reformulate representations, attention mechanisms, and optimization dynamics in terms of invariant relational quantities, eliminating redundant degrees of freedom by construction. This perspective yields architectures that operate directly on relational structures, providing a principled geometric framework for reducing parameter redundancy and analyzing optimization.

Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); High Energy Physics - Theory (hep-th); Machine Learning (stat.ML)

arXiv:2602.18948 [cs.LG]

(or arXiv:2602.18948v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2602.18948 Focus to learn more arXiv-issued DOI via DataCite

https://doi.org/10.1103/v57m-nhhq Focus to learn more DOI(s) linking to related resources

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.