aiqbench analysis
This research provides a rigorous theoretical framework demonstrating how two-layer neural networks spontaneously learn spectral representations to solve group composition tasks. By lifting training dynamics into the Fourier domain, the authors prove that neurons converge almost surely to specific irreducible representations while exhibiting novel low-rank compression properties in matrix-valued outputs. The work bridges representation theory and deep learning optimization, offering a mathematical account of feature emergence without relying on empirical heuristics.
Neural Networks Provably Learn Spectral Representations for Group Composition
arXiv:2606.02993v2 Announce Type: replace Abstract: Understanding how structured internal structure emerges during neural network training is central to the study of deep learning. We investigate this phenomenon through the group composition task, where a two-layer neural network is trained to predict $g_1 \star g_2$ for elements of a finite group $G$. By lifting the projected gradient flow to the Fourier domain, we demonstrate that the training dynamics are governed by a Riemannian gradient ascent on a representation-theoretic energy functional. We prove that, under random initialization, this flow drives each neuron to converge almost surely toward a single irreducible representation, while the cross-layer Fourier coefficients achieve a rotational rank-one alignment. This framework provides a representation-theoretic account of feature learning and characterizes a novel low-rank compression phenomenon for matrix-valued group representations. Moreover, for Abelian groups, we provide a complete population-level description: random initialization promotes uniform diversification across nontrivial representations and induces Haar-uniform phases, jointly approximating the indicator via a majority-vote mechanism. We further prove that both phase alignment and representation competition emerge with exponential convergence rates.
Abstract:Understanding how structured internal structure emerges during neural network training is central to the study of deep learning. We investigate this phenomenon through the group composition task, where a two-layer neural network is trained to predict $g_1 \star g_2$ for elements of a finite group $G$. By lifting the projected gradient flow to the Fourier domain, we demonstrate that the training dynamics are governed by a Riemannian gradient ascent on a representation-theoretic energy functional. We prove that, under random initialization, this flow drives each neuron to converge almost surely toward a single irreducible representation, while the cross-layer Fourier coefficients achieve a rotational rank-one alignment. This framework provides a representation-theoretic account of feature learning and characterizes a novel low-rank compression phenomenon for matrix-valued group representations. Moreover, for Abelian groups, we provide a complete population-level description: random initialization promotes uniform diversification across nontrivial representations and induces Haar-uniform phases, jointly approximating the indicator via a majority-vote mechanism. We further prove that both phase alignment and representation competition emerge with exponential convergence rates.
Machine Learning (cs.LG); Optimization and Control (math.OC); Representation Theory (math.RT); Statistics Theory (math.ST); Machine Learning (stat.ML)
arXiv:2606.02993 [cs.LG]
(or arXiv:2606.02993v2 [cs.LG] for this version)
Full text under the publisher's open licence · 193 words
Related coverage
Neural Networks Provably Learn Spectral Representations for Group Composition
arXiv · Jun 3, 2026
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
Introducing PIRAMID: Physics-Informed Research for Ambitious Mechanistic Interpretability
LessWrong · Jul 25, 2026
Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning
arXiv · Sep 7, 2026