aiqbench analysis
This study establishes that for equivariant neural networks, all non-polynomial activation functions like ReLU and sigmoid achieve equivalent maximum expressivity, rendering the choice between them theoretically neutral. Architectural depth improves separation power only up to a specific threshold, while adding invariant features to hidden layers has no impact on the model's ability to distinguish inputs. These findings provide a rigorous theoretical baseline for optimizing equivariant architectures without relying on empirical trial-and-error for these specific hyperparameters.
Separation Power of Equivariant Neural Networks
arXiv:2406.08966v3 Announce Type: replace-cross Abstract: The separation power of a machine learning model refers to its ability to distinguish between different inputs and is often used as a proxy for its expressivity. Indeed, knowing the separation power of a family of models is a necessary condition to obtain fine-grained universality results. In this paper, we analyze the separation power of equivariant neural networks, such as convolutional and permutation-invariant networks. We first present a complete characterization of inputs indistinguishable by models derived by a given architecture. From this results, we derive how separability is influenced by hyperparameters and architectural choices-such as activation functions, depth, hidden layer width, and representation types. Notably, all non-polynomial activations, including ReLU and sigmoid, are equivalent in expressivity and reach maximum separation power. Depth improves separation power up to a threshold, after which further increases have no effect. Adding invariant features to hidden representations does not impact separation power. Finally, block decomposition of hidden representations affects separability, with minimal components forming a hierarchy in separation power that provides a straightforward method for comparing the separation power of models.
Abstract:The separation power of a machine learning model refers to its ability to distinguish between different inputs and is often used as a proxy for its expressivity. Indeed, knowing the separation power of a family of models is a necessary condition to obtain fine-grained universality results. In this paper, we analyze the separation power of equivariant neural networks, such as convolutional and permutation-invariant networks. We first present a complete characterization of inputs indistinguishable by models derived by a given architecture. From this results, we derive how separability is influenced by hyperparameters and architectural choices-such as activation functions, depth, hidden layer width, and representation types. Notably, all non-polynomial activations, including ReLU and sigmoid, are equivalent in expressivity and reach maximum separation power. Depth improves separation power up to a threshold, after which further increases have no effect. Adding invariant features to hidden representations does not impact separation power. Finally, block decomposition of hidden representations affects separability, with minimal components forming a hierarchy in separation power that provides a straightforward method for comparing the separation power of models.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2406.08966 [cs.LG]
(or arXiv:2406.08966v3 [cs.LG] for this version)
Full text under the publisher's open licence · 190 words
Related coverage
GEqTrain: A Configuration-Driven Framework for Retargeting Equivariant Graph Neural Networks Across 3D Scientific Tasks
arXiv · Jul 22, 2026
On the Expressive Power of Sparse Geometric MPNNs
arXiv · Aug 3, 2026
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · Sep 18, 2026
IBM’s quantum GENCO neural solver unifies three grid tasks
Quantum Zeitgeist · Aug 12, 2026