aiqbench analysis
CAGE-NAS introduces a novel framework for efficient neural network growth by certifying functional gradient descent steps to determine when architectural expansion is necessary. This method avoids exhaustive search while achieving top-tier performance within fixed parameter budgets, offering a significant advance in scalable model training strategies.
CAGE-NAS: Certified Functional Descent for Efficient Model Growth
arXiv:2610.01173v1 Announce Type: new Abstract: The progressive growth of neural networks requires deciding when the current representation remains sufficient for optimization and when it should be expanded. CAGE-NAS formulates this decision in function space through an admissibility criterion on approximations of the functional gradient. As long as a representation enables a certified Functional Gradient Descent step, the architecture remains fixed; when the criterion fails, a function-preserving expansion is applied and the resulting representation is evaluated again. As the main instance, we study the family induced by the tangent space, using a regularized projection of the functional gradient. In a controlled setting with exact certification, CAGE-NAS produces architectures positioned above the 99.8th performance percentile by held-out RMSE among all admissible alternatives within the same parameter budget, without enumerating them during the growth trajectory.
Abstract:The progressive growth of neural networks requires deciding when the current representation remains sufficient for optimization and when it should be expanded. CAGE-NAS formulates this decision in function space through an admissibility criterion on approximations of the functional gradient. As long as a representation enables a certified Functional Gradient Descent step, the architecture remains fixed; when the criterion fails, a function-preserving expansion is applied and the resulting representation is evaluated again. As the main instance, we study the family induced by the tangent space, using a regularized projection of the functional gradient. In a controlled setting with exact certification, CAGE-NAS produces architectures positioned above the 99.8th performance percentile by held-out RMSE among all admissible alternatives within the same parameter budget, without enumerating them during the growth trajectory.
18 pages, 4 figures, 4 tables. Accepted at AXIOM 2026: Foundations of Efficient Deep Learning (NeurIPS 2026 Workshop)
Machine Learning (cs.LG)
arXiv:2610.01173 [cs.LG]
(or arXiv:2610.01173v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.01173 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 239 words
