aiqbench analysis
This research introduces Relative Generalization Invariance (RGI) as a novel phenomenon in LLM pretraining, demonstrating that optimizer and architecture choices induce uniform loss shifts while data streams significantly alter relative generalization. The study proves RGI emerges in overparameterized quadratic models beyond standard neural tangent kernel or mean-field regimes, offering a new lens to isolate the distinct effects of training components.
Relative Generalization Invariance of LLM Pretraining
arXiv:2609.33016v1 Announce Type: new Abstract: Large Language Model (LLM) pretraining performance is jointly shaped by three components of the training triplet: the optimizer, model architecture, and training data stream. However, how these components influence performance in distinct ways remains unclear. We take a first step toward isolating their effects by studying relative generalization. We introduce Relative Generalization Invariance (RGI), the invariance of the validation-loss difference between any two tokens across models. We show that RGI approximately holds across a wide range of optimizers and moderate architectural variations, suggesting that these choices induce an approximately uniform shift in token-wise losses. In contrast, changing the training data stream can substantially alter relative generalization. We further show that RGI cannot be explained by the neural tangent kernel or mean-field regimes alone and prove that it can emerge in an overparameterized quadratic model. Overall, our work identifies RGI as a new phenomenon in LLM pretraining that helps distinguish the effects of optimizers and architectures from those of training data.
Abstract:Large Language Model (LLM) pretraining performance is jointly shaped by three components of the training triplet: the optimizer, model architecture, and training data stream. However, how these components influence performance in distinct ways remains unclear. We take a first step toward isolating their effects by studying relative generalization. We introduce Relative Generalization Invariance (RGI), the invariance of the validation-loss difference between any two tokens across models. We show that RGI approximately holds across a wide range of optimizers and moderate architectural variations, suggesting that these choices induce an approximately uniform shift in token-wise losses. In contrast, changing the training data stream can substantially alter relative generalization. We further show that RGI cannot be explained by the neural tangent kernel or mean-field regimes alone and prove that it can emerge in an overparameterized quadratic model. Overall, our work identifies RGI as a new phenomenon in LLM pretraining that helps distinguish the effects of optimizers and architectures from those of training data.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2609.33016 [cs.LG]
(or arXiv:2609.33016v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.33016 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 257 words
Related coverage
Links #5: 2026/07
LessWrong · Jul 31, 2026
DR-VLA: Learning Human-Like Quadruped Navigation from Egocentric Web Videos via Domain-Invariant Scene Representations
arXiv · Oct 14, 2026
Frontier AI Shops Should Be Filtering and Generating AI Discourse During Pretraining
LessWrong · Sep 3, 2026
Prompt-Anchored Residual Adaptation for Biomedical Vision-Language Models
arXiv · Sep 30, 2026