aiqbench analysis
This research introduces the "effective alignment dimension" as a practical metric to predict whether expanding neural network width will improve performance on unseen data. By deriving finite-sample bounds that avoid restrictive spectral assumptions, the study provides actionable guidance for scaling LLMs and ResNets beyond traditional asymptotic theories. Experiments confirm this statistic reliably forecasts loss changes during residual expansion across major architectures like Pythia and Transformers.
Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension
arXiv:2607.24887v1 Announce Type: cross Abstract: Existing theories of neural-network width characterize asymptotic limits, but provide limited guidance on whether an expansion direction identified from finite training data remains beneficial on unseen data. We study this problem for function-preserving residual expansion and introduce the effective alignment dimension, a measurable quantity describing the signal-noise geometry of activation gradients. By deriving the exact mean and variance of the inner product between independently estimated training and test gradients, we obtain a finite-sample upper bound on misalignment probability. The bound depends only on the effective alignment dimension and an effective sample size, requiring finite second moments and a nonzero population gradient, without covariance spectral assumptions or prescribed width-growth rates. We integrate this certificate into the train-test residual-expansion framework, yielding a high-probability condition for test-risk improvement. Experiments across width-controlled LLaMA-style Transformers, Pythia, and ResNet-20 show that wider models exhibit larger effective alignment dimensions and lower empirical misalignment. Direct residual interventions confirm that the alignment statistic predicts the sign and magnitude of held-out loss changes.
Abstract:Existing theories of neural-network width characterize asymptotic limits, but provide limited guidance on whether an expansion direction identified from finite training data remains beneficial on unseen data. We study this problem for function-preserving residual expansion and introduce the effective alignment dimension, a measurable quantity describing the signal-noise geometry of activation gradients. By deriving the exact mean and variance of the inner product between independently estimated training and test gradients, we obtain a finite-sample upper bound on misalignment probability. The bound depends only on the effective alignment dimension and an effective sample size, requiring finite second moments and a nonzero population gradient, without covariance spectral assumptions or prescribed width-growth rates. We integrate this certificate into the train-test residual-expansion framework, yielding a high-probability condition for test-risk improvement. Experiments across width-controlled LLaMA-style Transformers, Pythia, and ResNet-20 show that wider models exhibit larger effective alignment dimensions and lower empirical misalignment. Direct residual interventions confirm that the alignment statistic predicts the sign and magnitude of held-out loss changes.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2607.24887 [cs.LG]
(or arXiv:2607.24887v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.24887 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 260 words
Related coverage
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
Challenge: Hand coding weights for efficient sequence memorisation
AI Alignment Forum · Jul 23, 2026
Generalization and infinite width
LessWrong · Aug 1, 2026
Evaluating task vectors, unlearning and inoculation
LessWrong · Sep 20, 2026