aiqbench analysis
This research introduces Neurogenesis Networks (NGN), a method that treats neural network size as a differentiable variable to be optimized directly during training rather than fixed beforehand. By learning the optimal number of structural components for various architectures like Transformers and MLPs, the study demonstrates that this dynamic approach yields performance comparable to standard fixed-size models while enabling efficient deployment through prefix selection.
NGN: Learning Neural Network Size as a Differentiable Count
arXiv:2609.27291v1 Announce Type: new Abstract: Neural network size is usually chosen before training, separating architecture selection from weight optimization. We introduce the Neurogenesis Network (NGN), a differentiable parameterization for learning how many ordered structural components a model should use. For each ordered component group, one learnable boundary selects an active prefix while the model parameters are trained. The boundary can grow from a compact initialization and can be deployed by discarding components beyond the learned boundary. Controlled experiments examine convergence of the learned boundary, the performance of deployed prefixes, and comparisons with fixed-size models and alternative approaches to learning capacity. We then apply the same mechanism to MLPs, convolutional and graph networks, Transformers, state-space models, LoRA, and adapters. Across these settings, deploying only the learned prefix usually changes performance little, and the selected architectures perform similarly to fixed models trained at the same size. These results show that structural capacity can be optimized directly as a count.
Abstract:Neural network size is usually chosen before training, separating architecture selection from weight optimization. We introduce the Neurogenesis Network (NGN), a differentiable parameterization for learning how many ordered structural components a model should use. For each ordered component group, one learnable boundary selects an active prefix while the model parameters are trained. The boundary can grow from a compact initialization and can be deployed by discarding components beyond the learned boundary. Controlled experiments examine convergence of the learned boundary, the performance of deployed prefixes, and comparisons with fixed-size models and alternative approaches to learning capacity. We then apply the same mechanism to MLPs, convolutional and graph networks, Transformers, state-space models, LoRA, and adapters. Across these settings, deploying only the learned prefix usually changes performance little, and the selected architectures perform similarly to fixed models trained at the same size. These results show that structural capacity can be optimized directly as a count.
Machine Learning (cs.LG)
arXiv:2609.27291 [cs.LG]
(or arXiv:2609.27291v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.27291 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 246 words
Related coverage
Dropless MoE Training in JAX with Primus-Turbo
AMD · Jun 10, 2026
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
DiffCoord: Differentiable Coordination for Distributed Multi-Agent Trajectory Optimization
arXiv · Jun 12, 2026
A Convex Quasilinearization Method for Solving Nonlinear PDEs with Physics-Informed Neural Networks
arXiv · Jun 17, 2026