analiza aiqbench
This study investigates how to optimally allocate fixed parameter budgets between network depth and width by analyzing gradient superposition mechanisms. The authors demonstrate that error-response amplification serves as a key link between this geometric property and model generalization, offering a functional account for efficient parameter utilization in large language models.
The Model Shape Behind Scaling Laws: A Gradient Superposition Perspective
arXiv:2605.27989v2 Announce Type: replace Abstract: Neural scaling laws guide resource allocation, but how should a fixed parameter budget be divided between depth and width? Motivated by inverse-width representation-error scaling under strong superposition, we study this allocation through the network's end-to-end response. The Neural Feature Ansatz links weight and gradient geometry, motivating the Frobenius norm $F$ of the input average gradient outer product (AGOP) as an observable of response strength and overlap. By reproducing Anthropic's toy-model double-descent experiment, we find that $F$ and test reconstruction loss follow an approximately affine relationship across the full sample-size sweep and share an interpolation peak. Through Gaussian minimum-norm regression experiments, we further show that this correlation arises from a fitted-noise amplification term shared by prediction error and $F$. By pretraining language models spanning different shapes and parameter budgets, we find a strong positive correlation between loss and $F$ among the best shapes selected by validation loss. This supports interpreting effective parameter allocation as preserving task-relevant computation while reducing the amplification of error responses. Our results identify error-response amplification as a mechanism linking gradient superposition to generalization, providing a functional account of depth-width allocation and efficient parameter utilization. The code is availale at: $\hyperlink{https://github.com/wenjie1835/Shape_Scaling}{https://github.com/wenjie1835/Shape_Scaling}$.
Abstract:Neural scaling laws guide resource allocation, but how should a fixed parameter budget be divided between depth and width? Motivated by inverse-width representation-error scaling under strong superposition, we study this allocation through the network's end-to-end response. The Neural Feature Ansatz links weight and gradient geometry, motivating the Frobenius norm $F$ of the input average gradient outer product (AGOP) as an observable of response strength and overlap. By reproducing Anthropic's toy-model double-descent experiment, we find that $F$ and test reconstruction loss follow an approximately affine relationship across the full sample-size sweep and share an interpolation peak. Through Gaussian minimum-norm regression experiments, we further show that this correlation arises from a fitted-noise amplification term shared by prediction error and $F$. By pretraining language models spanning different shapes and parameter budgets, we find a strong positive correlation between loss and $F$ among the best shapes selected by validation loss. This supports interpreting effective parameter allocation as preserving task-relevant computation while reducing the amplification of error responses. Our results identify error-response amplification as a mechanism linking gradient superposition to generalization, providing a functional account of depth-width allocation and efficient parameter utilization. The code is availale at: $\hyperlink{this https URL}{this https URL}$.
Machine Learning (cs.LG)
arXiv:2605.27989 [cs.LG]
(or arXiv:2605.27989v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.27989 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 289 cuvinte
