analiza aiqbench
This research investigates the relationship between training loss and Fisher width, demonstrating how gradient descent trajectories asymptotically select specific branches of parameter space in Gaussian-teacher models. The study provides theoretical bounds and experimental validation showing that optimization algorithms like SGD and Adam follow distinct paths relative to this geometric measure, challenging the notion of a universal loss parametrization. These findings offer critical insights into the geometric structure of learning dynamics and the stability of probe sets during model training.
Loss-Parameterized Fisher Width Along Learning Trajectories
arXiv:2608.21561v1 Announce Type: new Abstract: Fisher width measures the Gaussian width of a probe set after deformation by the local Fisher geometry. We study its evolution along learning trajectories and ask when training loss can serve as an effective coordinate for this quantity. We first derive an exact trace--shape factorization and a deterministic stability bound for fixed compact probes. In a population Gaussian-teacher logistic model, the teacher-aligned state is extremal on every loss level below $\log 2$: it has minimal parameter norm and maximizes both Fisher trace and Euclidean-ball Fisher width. We then show that population gradient flow asymptotically selects this branch, with explicit rates for the aligned and orthogonal coordinates. This yields, for $d\geq2$, \[ \frac{w_F(B_2^d;\theta(t))} {\sqrt{L(\theta(t))}} \longrightarrow \frac{\sqrt6}{\pi}\mathbb E[\chi_{d-1}]. \] Controlled full-Fisher experiments support the matched-loss branch and the population predictions. In a nonlinear MLP with a diagonal model-Fisher approximation, GD and SGD remain close at matched loss, whereas Adam follows a substantially displaced branch; the fixed probes tested retain highly similar temporal shapes. These results support a branchwise, rather than universal, loss parametrization of Fisher width.
Abstract:Fisher width measures the Gaussian width of a probe set after deformation by the local Fisher geometry. We study its evolution along learning trajectories and ask when training loss can serve as an effective coordinate for this quantity. We first derive an exact trace--shape factorization and a deterministic stability bound for fixed compact probes. In a population Gaussian-teacher logistic model, the teacher-aligned state is extremal on every loss level below $\log 2$: it has minimal parameter norm and maximizes both Fisher trace and Euclidean-ball Fisher width. We then show that population gradient flow asymptotically selects this branch, with explicit rates for the aligned and orthogonal coordinates. This yields, for $d\geq2$, \[ \frac{w_F(B_2^d;\theta(t))} {\sqrt{L(\theta(t))}} \longrightarrow \frac{\sqrt6}{\pi}\mathbb E[\chi_{d-1}]. \] Controlled full-Fisher experiments support the matched-loss branch and the population predictions. In a nonlinear MLP with a diagonal model-Fisher approximation, GD and SGD remain close at matched loss, whereas Adam follows a substantially displaced branch; the fixed probes tested retain highly similar temporal shapes. These results support a branchwise, rather than universal, loss parametrization of Fisher width.
Machine Learning (cs.LG); Statistics Theory (math.ST)
arXiv:2608.21561 [cs.LG]
(or arXiv:2608.21561v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.21561 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 269 cuvinte
Subiecte conexe
Large to Small Model Stitching Destroys What You Built It to Carry Along the Fit
LessWrong · 11 oct. 2026
When Partial Credit Opposes Completion
Snorkel AI Blog · 15 oct. 2026
Evaluating task vectors, unlearning and inoculation
LessWrong · 20 sept. 2026
An Energy-Based Conservative-Dissipative Latent Neural Evolution Operator for Magnetization Dynamics
arXiv · 7 sept. 2026