analiza aiqbench
This research extends the theoretical understanding of gradient descent dynamics by establishing conservation laws for modern neural architectures beyond simple linear or ReLU models. The study provides a unified framework that characterizes these invariants across contemporary designs, including SwiGLU activations and Mixture-of-Experts systems, validating predictions through empirical experiments. Such insights are crucial for explaining the implicit biases of over-parameterized models as they scale to increasingly complex structures.
Conservation Laws for Modern Neural Architectures
arXiv:2606.17816v1 Announce Type: new Abstract: Understanding gradient descent dynamics is key to explaining the success of over-parameterized models, where implicit bias manifests through conservation laws in gradient flow. While such laws are well understood for linear and ReLU networks, they remain largely unexplored for modern architectures. This work develops a unified framework to characterize conservation laws for contemporary models, including feedforward networks with GELU, SiLU, and SwiGLU activations, multihead attention with sinusoidal and rotary positional encodings, and Mixture-of-Experts architectures under diverse gating designs. Our theoretical findings are supported by experiments that validate the predicted invariants.
Abstract:Understanding gradient descent dynamics is key to explaining the success of over-parameterized models, where implicit bias manifests through conservation laws in gradient flow. While such laws are well understood for linear and ReLU networks, they remain largely unexplored for modern architectures. This work develops a unified framework to characterize conservation laws for contemporary models, including feedforward networks with GELU, SiLU, and SwiGLU activations, multihead attention with sinusoidal and rotary positional encodings, and Mixture-of-Experts architectures under diverse gating designs. Our theoretical findings are supported by experiments that validate the predicted invariants.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2606.17816 [cs.LG]
(or arXiv:2606.17816v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.17816 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 187 cuvinte
Subiecte conexe
The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints
arXiv · 16 iun. 2026
Spider - Man : Brand New Day Wearable AI Cannot Exist Yet : Real Power - Budget Wall Stands in Its Way
techtimes.com · 3 aug. 2026
AI Futures: Racing to Lose, A sermon
LessWrong · 28 sept. 2026
PIRAMID: Progress and Plans
LessWrong · 27 iul. 2026