aiqbench analysis
The paper proposes activation functions built around chosen equivariances, so that dense layers can be diagonalised via SVD and neurons can be added or pruned while preserving function. Its claims of exact function preservation under neurogenesis, and of asymptotic 50% sparsification, are analytical, and the abstract gives no benchmark results, so readers should look for empirical validation before treating the adaptive-topology approach as practical. The idea of symmetry-led, basis-independent neuron design is still worth tracking for dynamic architecture work.
Isotropic Activation Functions Enable Deindividuated Neurons and Adaptive Topologies
arXiv:2602.23405v3 Announce Type: replace-cross Abstract: Varying the number of neurons while preserving function is difficult because neural networks' dense connectivity creates strong computational interdependence between units. Elementwise activation functions, particularly their limited equivariance properties, are identified as a central hurdle to achieving this. We propose activation functions designed around chosen equivariances, enabling dense neural networks to adapt via singular value decomposition of their affine maps. This diagonalises layers into one-to-one, ordered connections, making it easier to assess the impact of individual connections on function. Low-impact neurons can be removed (neurodegeneration), and a thresholded buffer of largely inactive `scaffold' neurons is maintained (neurogenesis). These symmetry-led diagonalisation and structural changes prioritise function-invariance: neurogenesis exactly preserves function, neurodegeneration is asymptotically function-preserving with the pruned singular-value threshold. Moreover, this construction enables asymptotic 50\% parameter sparsification of dense networks while preserving function analytically. Thus, the approach enables real-time restructuring of the architecture in response to task demands, task appending, removal or changes. Conceptually, this motivates broader applications of symmetry prescriptions at the small scale, even when no data-driven model-scale symmetry adherence is required. Following this, we build activations around symmetry, not from classical assumptions about individual neurons. Unlike typical elementwise forms, these alternatives feature explicit basis independence and do not individuate neurons. We leverage this to decompose and reinterpret neurons across sequential layers, simplifying structure and providing a natural route to adaptive topologies. Additional developments include a new tunable parameter, the `intrinsic length'.
Abstract:Varying the number of neurons while preserving function is difficult because neural networks' dense connectivity creates strong computational interdependence between units. Elementwise activation functions, particularly their limited equivariance properties, are identified as a central hurdle to achieving this. We propose activation functions designed around chosen equivariances, enabling dense neural networks to adapt via singular value decomposition of their affine maps. This diagonalises layers into one-to-one, ordered connections, making it easier to assess the impact of individual connections on function. Low-impact neurons can be removed (neurodegeneration), and a thresholded buffer of largely inactive `scaffold' neurons is maintained (neurogenesis). These symmetry-led diagonalisation and structural changes prioritise function-invariance: neurogenesis exactly preserves function, neurodegeneration is asymptotically function-preserving with the pruned singular-value threshold. Moreover, this construction enables asymptotic 50\% parameter sparsification of dense networks while preserving function analytically. Thus, the approach enables real-time restructuring of the architecture in response to task demands, task appending, removal or changes. Conceptually, this motivates broader applications of symmetry prescriptions at the small scale, even when no data-driven model-scale symmetry adherence is required. Following this, we build activations around symmetry, not from classical assumptions about individual neurons. Unlike typical elementwise forms, these alternatives feature explicit basis independence and do not individuate neurons. We leverage this to decompose and reinterpret neurons across sequential layers, simplifying structure and providing a natural route to adaptive topologies. Additional developments include a new tunable parameter, the `intrinsic length'.
Neural and Evolutionary Computing (cs.NE); Machine Learning (cs.LG)
I.2.6; I.2.0; G.1.3
arXiv:2602.23405 [cs.NE]
(or arXiv:2602.23405v3 [cs.NE] for this version)
https://doi.org/10.48550/arXiv.2602.23405 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 333 words