Thoughts on 'Learning Mechanics'

Last month, I stumbled upon an optimistic-sounding paper: "There Will Be a Scientific Theory of Deep Learning". In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. Historically, machine learning has been a field where theoretical understanding has lagged far behind empirical success. This paper argues that might soon start to change. The authors propose that we are gradually seeing the emergence of what they call learning mechanics---a mathematical understanding of the process by which machine learning models learn. And the newly-developing theory looks an awful lot like physics! We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We assert that learning mechanics should be a mathematical theory, grounded in first-principles calculations that closely predict empirics, reliant on well-tested approximations and assumptions, aiming for broad impact across the machine learning stack once it reaches maturity. The name "learning mechanics" was deliberately chosen to echo the various mechanics of physics. Examples of mechanics are: Classical mechanics: the study of how objects move and interact under the influence of forces. Statistical mechanics: the study of how the collective behavior of many microscopic particles gives rise to macroscopic phenomena like temperature and pressure. Quantum mechanics: the study of how matter and energy behave at the smallest scales, where physical quantities are quantized and governed by probabilistic laws. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions. There is a clear emphasis on the dynamics of the learning process. Others in machine learning share this perspective. For example, at the most recent NeurIPS, there was a DynaFront workshop focused on applications of dynamical systems theory to machine learning. There is also a clear emphasis on the importance of empirics. Historically, many theoretical machine learning researchers have come from computer science backgrounds, where the research style is much closer to math than to physics: an emphasis on formal proofs and rigor. While rigor is valuable, there has perhaps been an underemphasis on mathematical models that, while not entirely rigorous by the standards of proof-based mathematics, are motivated by deep principles and can explain important, previously-unexplained aspects of real-world data. The authors do offer a note of caution though: learning mechanics is not meant to be the be-all and end-all. They acknowledge that different levels of description will always be necessary. Just as we still need biology to understand the anatomy of organisms and psychology to usefully model human behavior, there will be analogous magisteria for understanding neural networks. They describe their approach as complementary to alternative perspectives like mechanistic interpretability. We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic and mutually supportive relationship between learning mechanics and the developing discipline of mechanistic interpretability. Where mechanistic interpretability aims to be the biology of deep learning, learning mechanics should aspire to be its physics, mirroring the complementary relationship between biology and physics in the natural sciences. Putting my cards on the table: I really like the framing of learning mechanics. My current work falls squarely under the umbrella they describe. The idea that my research is not an isolated effort but part of the broader emergence of something as estimable as a Learning Mechanics™ seems tailored to flatter my biases. In addition to releasing the paper, the authors launched a website which features essays that build up the learning mechanics framework. There is also a podcast episode where Jamie Simon and Daniel Kunin, the two lead co-authors of the paper, discuss the ideas it touches on. The Five Approaches One could ask: Why should we think a theory of deep learning is even possible? A great cause for optimism that a mechanics of learning is possible is the fact that the essential ingredients of deep learning are both explicit and measurable. Nothing about the learning process is hidden. Unlike many complex systems where the equations governing dynamics must be inferred from observations, deep learning directly exposes its “equations of motion.” Moreover, these dynamics are extraordinarily measurable: