analiza aiqbench
This paper provides a rigorous mathematical framework for auxiliary learning, deriving closed-form expressions that quantify how task correlations and label noise influence generalization error in neural networks. By modeling the dynamics of stochastic gradient descent through differential equations, the study reveals the specific mechanisms by which auxiliary tasks balance optimization forces with gradient noise to improve performance. These theoretical insights offer a deeper understanding of multi-task optimization beyond empirical observation, directly advancing machine learning theory.
An Analytical Theory of Auxiliary Learning
arXiv:2609.29774v1 Announce Type: new Abstract: Auxiliary learning is an optimization paradigm in which a neural network's performance on a target task is improved by jointly training it on additional tasks. However, the mechanisms behind this improvement remain poorly understood. We study this problem using a teacher-student framework and derive a closed system of differential equations describing the dynamics of online stochastic gradient descent in the large-input limit. For linear networks, we obtain a closed-form expression for the generalization error to leading order in the learning rate, quantifying how task correlations and label noise determine the benefit of auxiliary learning. For non-linear activation functions, we develop a fluctuation-dissipation analytical theory that establishes a general relation linking the main and auxiliary errors to the corresponding single-task error. Numerical experiments support the theoretical predictions and show how auxiliary tasks improve generalization by balancing the forcing dynamics towards the optimal solution with gradient noise.
Abstract:Auxiliary learning is an optimization paradigm in which a neural network's performance on a target task is improved by jointly training it on additional tasks. However, the mechanisms behind this improvement remain poorly understood. We study this problem using a teacher-student framework and derive a closed system of differential equations describing the dynamics of online stochastic gradient descent in the large-input limit. For linear networks, we obtain a closed-form expression for the generalization error to leading order in the learning rate, quantifying how task correlations and label noise determine the benefit of auxiliary learning. For non-linear activation functions, we develop a fluctuation-dissipation analytical theory that establishes a general relation linking the main and auxiliary errors to the corresponding single-task error. Numerical experiments support the theoretical predictions and show how auxiliary tasks improve generalization by balancing the forcing dynamics towards the optimal solution with gradient noise.
Machine Learning (cs.LG)
arXiv:2609.29774 [cs.LG]
(or arXiv:2609.29774v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.29774 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 239 cuvinte
Subiecte conexe
Learning the Kohn-Sham map with neural operators for quasi-linear scaling density functional theory
arXiv · 27 aug. 2026
Engineering exact mobility edges in quasiperiodic Aharonov-Bohm chains
arXiv · 25 aug. 2026
Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers
arXiv · 24 iul. 2026
Gravity from generalized entropy
arXiv · 15 oct. 2026