From Objectives to What Models Learn: A Landau Theory of Invariant Learning

Abstract:Invariant-learning objectives pursue similar goals yet produce qualitatively different regularization paths, leaving unclear when shortcuts can be suppressed without damaging stable modes. Our starting point is simple: in the desired shortcut-suppressed regime, shortcut loading is small, so the objective's low-order expansion governs local stability and residual amplitude. This brings the problem into the domain of Landau phase-transition theory. We establish a mathematical isomorphism between the near-critical normal form of predictive-mode learning and Landau theory, identifying the learning objective as an effective free energy and critical-mode amplitude as an order parameter. The resulting low-order objective signatures predict distinct regularization phenotypes: quadratic $R_2$ terms shift phase boundaries and enable finite-strength elimination, whereas quartic $R_4$ terms continuously attenuate acquired modes while leaving nonzero residual loading at finite strengths. Higher-order terms may further shape nonlinear tails. In a canonical bilinear model, we derive exact phase boundaries and equilibrium loadings, identifying conditions for a selective-retention window in which shortcut suppression preserves stable structure. Controlled bilinear and ReLU experiments, including an MNIST construction, support the predicted signature-phenotype relation, while coupled-feature experiments validate its extension to collective modes. The framework connects the mathematical structure of invariant-learning objectives to what models learn as regularization varies.

Machine Learning (cs.LG)

arXiv:2608.09396 [cs.LG]

(or arXiv:2608.09396v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2608.09396 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.