aiqbench analysis
The paper shows that Transformers with identical functions can diverge in training because reparameterizations like rescaling query and key matrices change gradient scales that standard optimizers, including AdamW, don't correct. Its proposed SAGGS method equalizes Frobenius norms of symmetry-paired matrices, with reported perplexity gains of 0.37 to 0.78 on WikiText-103 and a 31M-parameter replication. For ML practitioners, the practical takeaway is that checkpoint resumption and optimizer behavior can depend on parameterization details, though the gains are modest and need independent replication at larger scale.
Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization
arXiv:2610.18085v1 Announce Type: new Abstract: We show that two Transformers representing the same function can follow different training trajectories. For example, doubling a query matrix and halving its key matrix preserves the model's function but changes the gradient scales seen by the optimizer. Our theoretical analysis establishes that standard block-local first-order optimizers fail to scale updates with parameters under a shared learning rate. For AdamW, transforming the first- and second-moment estimates alone does not correct the mismatch between parameter and update scales. To make runs restarted from equivalent checkpoints follow the same trajectory, we balance matrix pairs and transform their moment estimates together, bringing distinct but equivalent training states to the same starting point. To correct scale imbalance between paired Transformer matrices without changing predictions, we propose Symmetry-Aware Gradient Geometry Stabilization (SAGGS), which periodically equalizes the Frobenius norms of exact symmetry pairs, and its damped variant, Soft SAGGS. SAGGS reduces perplexity (PPL) by $0.374\pm0.026$ at 101M parameters on WikiText-103, and Soft SAGGS reduces PPL by $0.780\pm0.040$ in a 31M modern-architecture replication, both under scale imbalance. In a 123.8M checkpoint-resume experiment, reciprocally rescaling matrix pairs barely changes validation loss at restart but increases final PPL by $3.78\pm0.42$; hard and soft rebalancing recover to within $0.02$ PPL of the control.
Abstract:We show that two Transformers representing the same function can follow different training trajectories. For example, doubling a query matrix and halving its key matrix preserves the model's function but changes the gradient scales seen by the optimizer. Our theoretical analysis establishes that standard block-local first-order optimizers fail to scale updates with parameters under a shared learning rate. For AdamW, transforming the first- and second-moment estimates alone does not correct the mismatch between parameter and update scales. To make runs restarted from equivalent checkpoints follow the same trajectory, we balance matrix pairs and transform their moment estimates together, bringing distinct but equivalent training states to the same starting point. To correct scale imbalance between paired Transformer matrices without changing predictions, we propose Symmetry-Aware Gradient Geometry Stabilization (SAGGS), which periodically equalizes the Frobenius norms of exact symmetry pairs, and its damped variant, Soft SAGGS. SAGGS reduces perplexity (PPL) by $0.374\pm0.026$ at 101M parameters on WikiText-103, and Soft SAGGS reduces PPL by $0.780\pm0.040$ in a 31M modern-architecture replication, both under scale imbalance. In a 123.8M checkpoint-resume experiment, reciprocally rescaling matrix pairs barely changes validation loss at restart but increases final PPL by $3.78\pm0.42$; hard and soft rebalancing recover to within $0.02$ PPL of the control.
Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
arXiv:2610.18085 [cs.LG]
(or arXiv:2610.18085v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.18085 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 302 words
Related coverage
Optimization Geometry of Equivalent Brownian RKHS Representations
arXiv · Sep 21, 2026
The Entangled Dimensions of Decision Theory
LessWrong · Jul 30, 2026
Academia is for Ambition — Alex Zhang, MIT
Latent Space · Oct 2, 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · Sep 29, 2026