analiza aiqbench
This study demonstrates that adding parameters via model growth creates distinct, persistent functional directions even when a model catastrophically forgets its original tasks. The research proves that geometric capacity and retained function are decoupled, meaning increasing parameter count alone does not guarantee the preservation of previously learned capabilities. These findings offer critical insights for scaling foundation models without assuming linear gains in effective knowledge retention.
What Does Model Growth Really Add? Functional Capacity Beyond Parameter Count
arXiv:2607.14571v2 Announce Type: replace Abstract: Model growth adds parameters to a trained checkpoint and continues training, aiming to increase capacity while reusing the existing model instead of training from scratch. But does growing a trained model actually create new functional capacity that survives subsequent training? Parameter count alone cannot answer this question, and ``capacity'' is itself ambiguous; it can refer to parameter count, effective dimensionality, or the ability to represent new functions. We measure it directly, as the number of new, functionally independent directions a growth step adds, formalized as the rank increment of the model's functional Jacobian and estimated matrix-free at scale. For zero-gated function-preserving growth, this increment is exactly $K$ under a transversality condition, i.e. $\operatorname{rank}(J_{\mathrm{grown}})=\operatorname{rank}(J_{\mathrm{old}})+K$. Continually training a Transformer grown from $253\mathrm{M}$ to $857\mathrm{M}$ parameters across three domains, we find these added directions persist, with all $K=36$ remaining independent after continual learning, even when the model catastrophically forgets (perplexity $28\to481$) or we deliberately destroy the old function ($25.9\to127$). Geometric capacity and retained function are therefore decoupled. The functional capacity introduced by growth persists, while the previously learned function is not retained. A second, geometrically distinct mechanism, Net2Net, reaches the same $K$-dimensional capacity from directions that are dormant at initialization ($0 \to K$) and then retains it, so the decoupling is not specific to the zero-gate parameterization. The implication is precise. Catastrophic forgetting need not reflect a loss of functional capacity, and preserving dimensionality alone is insufficient to preserve the learned function.
Title:What Does Model Growth Really Add? Functional Capacity Beyond Parameter Count
Abstract:Model growth adds parameters to a trained checkpoint and continues training, aiming to increase capacity while reusing the existing model instead of training from scratch. But does growing a trained model actually create new functional capacity that survives subsequent training? Parameter count alone cannot answer this question, and ``capacity'' is itself ambiguous; it can refer to parameter count, effective dimensionality, or the ability to represent new functions. We measure it directly, as the number of new, functionally independent directions a growth step adds, formalized as the rank increment of the model's functional Jacobian and estimated matrix-free at scale. For zero-gated function-preserving growth, this increment is exactly $K$ under a transversality condition, i.e. $\operatorname{rank}(J_{\mathrm{grown}})=\operatorname{rank}(J_{\mathrm{old}})+K$. Continually training a Transformer grown from $253\mathrm{M}$ to $857\mathrm{M}$ parameters across three domains, we find these added directions persist, with all $K=36$ remaining independent after continual learning, even when the model catastrophically forgets (perplexity $28\to481$) or we deliberately destroy the old function ($25.9\to127$). Geometric capacity and retained function are therefore decoupled. The functional capacity introduced by growth persists, while the previously learned function is not retained. A second, geometrically distinct mechanism, Net2Net, reaches the same $K$-dimensional capacity from directions that are dormant at initialization ($0 \to K$) and then retains it, so the decoupling is not specific to the zero-gate parameterization. The implication is precise. Catastrophic forgetting need not reflect a loss of functional capacity, and preserving dimensionality alone is insufficient to preserve the learned function.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2607.14571 [cs.LG]
(or arXiv:2607.14571v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.14571 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 345 cuvinte
Subiecte conexe
Netskope (NTSK) Q2 2027 Earnings Call Transcript
fool.com · 10 sept. 2026
Microsoft’s AI chief says superintelligence is near, but won’t take your job
The Verge · 8 iun. 2026
Quantum Computing (QUBT) Q2 2026 Earnings Call Transcript
fool.com · 18 aug. 2026
Links #3: 2026/06 Part 1
LessWrong · 15 iun. 2026