Neural Networks Provably Learn Spectral Representations for Group Composition

Abstract:Understanding how structured internal structure emerges during neural network training is central to the study of deep learning. We investigate this phenomenon through the group composition task, where a two-layer neural network is trained to predict $g_1 \star g_2$ for elements of a finite group $G$. By lifting the projected gradient flow to the Fourier domain, we demonstrate that the training dynamics are governed by a Riemannian gradient ascent on a representation-theoretic energy functional. We prove that, under random initialization, this flow drives each neuron to converge almost surely toward a single irreducible representation, while the cross-layer Fourier coefficients achieve a rotational rank-one alignment. This framework provides a representation-theoretic account of feature learning and characterizes a novel low-rank compression phenomenon for matrix-valued group representations. Moreover, for Abelian groups, we provide a complete population-level description: random initialization promotes uniform diversification across nontrivial representations and induces Haar-uniform phases, jointly approximating the indicator via a majority-vote mechanism. We further prove that both phase alignment and representation competition emerge with exponential convergence rates.

Machine Learning (cs.LG); Optimization and Control (math.OC); Representation Theory (math.RT); Statistics Theory (math.ST); Machine Learning (stat.ML)

arXiv:2606.02993 [cs.LG]

(or arXiv:2606.02993v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2606.02993 arXiv-issued DOI via DataCite (pending registration)