aiqbench analysis
This paper gives a unified spectral account of maximal update parameterization (μP) for networks scaled in both width and depth, and argues that the CompleteP-style scheme suits Transformer-like residual blocks with multiple transformations. For ML practitioners, the practical value is reliable hyperparameter transfer from small proxy models to large GPT-2-style language models, though the GPT-2 experiments are the main evidence and broader validation is still needed.
Spectral Condition for $\mu$P under Width-Depth Scaling
arXiv:2603.00541v3 Announce Type: replace Abstract: Generative foundation models are increasingly scaled in both width and depth, posing significant challenges for stable feature learning and reliable hyperparameter (HP) transfer across model sizes. While maximal update parameterization~($\mu$P) has provided a principled solution to both problems for width scaling, existing extensions to the joint width-depth scaling regime remain fragmented, architecture- and optimizer-specific, and often rely on technically involved theories. In this work, we develop a simple and unified spectral framework for $\mu$P under joint width-depth scaling. For deep residual networks whose residual blocks contain $k$ transformations, the framework specifies how the norms of weights and their per-step updates should scale with width and depth. It reveals a fundamental transition from $k=1$ to $k\geq 2$, {unifying previously disparate $\mu$P formulations and identifying the CompleteP-style parameterization ($k\geq 2$)} as more appropriate for practical architectures with multi-transformation branches such as Transformers. Building on this framework, we derive a general recipe for implementing $\mu$P across a broad class of optimizers by mapping spectral constraints to {concrete, optimizer-specific HP parameterizations}, recovering existing results and extending them to additional optimizers. {Finally, experiments on GPT-2 style language models show that the CompleteP-style parameterization ($k\geq 2$) achieves stable feature learning and robust HP transfer under width-depth scaling, whereas standard parameterization and the Depth-$\mu$P-style parameterization ($k=1$) often fail to do so.} These results support the practical effectiveness of the proposed spectral framework.
Abstract:Generative foundation models are increasingly scaled in both width and depth, posing significant challenges for stable feature learning and reliable hyperparameter (HP) transfer across model sizes. While maximal update parameterization~($\mu$P) has provided a principled solution to both problems for width scaling, existing extensions to the joint width-depth scaling regime remain fragmented, architecture- and optimizer-specific, and often rely on technically involved theories. In this work, we develop a simple and unified spectral framework for $\mu$P under joint width-depth scaling. For deep residual networks whose residual blocks contain $k$ transformations, the framework specifies how the norms of weights and their per-step updates should scale with width and depth. It reveals a fundamental transition from $k=1$ to $k\geq 2$, {unifying previously disparate $\mu$P formulations and identifying the CompleteP-style parameterization ($k\geq 2$)} as more appropriate for practical architectures with multi-transformation branches such as Transformers. Building on this framework, we derive a general recipe for implementing $\mu$P across a broad class of optimizers by mapping spectral constraints to {concrete, optimizer-specific HP parameterizations}, recovering existing results and extending them to additional optimizers. {Finally, experiments on GPT-2 style language models show that the CompleteP-style parameterization ($k\geq 2$) achieves stable feature learning and robust HP transfer under width-depth scaling, whereas standard parameterization and the Depth-$\mu$P-style parameterization ($k=1$) often fail to do so.} These results support the practical effectiveness of the proposed spectral framework.
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2603.00541 [cs.LG]
(or arXiv:2603.00541v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2603.00541 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 320 words