analiza aiqbench
This paper bridges singular learning theory and information geometry by deriving the KL divergence order directly from the decay rate of the directional Fisher metric near analytic singularities, bypassing traditional resolution methods. The authors extend this framework to deep networks using K-FAC factorization to establish a duality between activation and gradient rates, ultimately proposing DDCAdam, an equivariant optimizer that commutes with group actions to preserve these theoretical guarantees.
Dead Directions: Geometric Singular Learning
arXiv:2606.05957v2 Announce Type: replace Abstract: Singular learning theory and information geometry study the same spaces: the former in resolved coordinates, the latter in original coordinates under a non-degeneracy assumption that overparameterised models violate. This paper carries one direction of the bridge between them, from Watanabe's invariants to Fisher geometry, through one primitive, the dead direction: a unit vector along which the Fisher metric degenerates, equivalently a direction crossing the analytic singular set along which the KL divergence keeps a zero of high order, its KL order set by how fast that divergence vanishes. Our central result recovers the KL order as the decay rate of the directional Fisher quadratic form approaching the singularity, in original coordinates, without a Hironaka resolution. A selection rule on smooth fibres translates this rate into Watanabe's single-direction contribution to the real log canonical threshold, and the recovery extends to multi-component crossings, multiplicity $m$, the singular fluctuation $\nu$, prior-RLCT shifts, and tempered posteriors. We then carry the rate into a deep network: a multi-layer K-FAC factorisation writes each Fisher block as a product of activation- and gradient-side rates with a duality between them, instantiated at residual streams, layer normalisation, and attention. A quotient theorem carries the rate to the gauge quotient for optimizers whose update commutes with the group action; Adam's per-coordinate preconditioner fails that condition, so we construct DDCAdam, an equivariant Adam-family preconditioner, and prove the quotient rate along its trajectory. The result is a trajectory-rate readout of Watanabe's triple $(\lambda, m, \nu)$ from one checkpoint's forward and backward passes, without posterior sampling.
Abstract:Singular learning theory and information geometry study the same spaces: the former in resolved coordinates, the latter in original coordinates under a non-degeneracy assumption that overparameterised models violate. This paper carries one direction of the bridge between them, from Watanabe's invariants to Fisher geometry, through one primitive, the dead direction: a unit vector along which the Fisher metric degenerates, equivalently a direction crossing the analytic singular set along which the KL divergence keeps a zero of high order, its KL order set by how fast that divergence vanishes. Our central result recovers the KL order as the decay rate of the directional Fisher quadratic form approaching the singularity, in original coordinates, without a Hironaka resolution. A selection rule on smooth fibres translates this rate into Watanabe's single-direction contribution to the real log canonical threshold, and the recovery extends to multi-component crossings, multiplicity $m$, the singular fluctuation $\nu$, prior-RLCT shifts, and tempered posteriors. We then carry the rate into a deep network: a multi-layer K-FAC factorisation writes each Fisher block as a product of activation- and gradient-side rates with a duality between them, instantiated at residual streams, layer normalisation, and attention. A quotient theorem carries the rate to the gauge quotient for optimizers whose update commutes with the group action; Adam's per-coordinate preconditioner fails that condition, so we construct DDCAdam, an equivariant Adam-family preconditioner, and prove the quotient rate along its trajectory. The result is a trajectory-rate readout of Watanabe's triple $(\lambda, m, \nu)$ from one checkpoint's forward and backward passes, without posterior sampling.
v2: substantially revised. 176 pages, 13 figures, 14 tables. New machine-checked appendix (Lean 4 sources as ancillary files); singular-fluctuation theorem restated and its strict inequality proved; determinantal RLCT closed at depth <= 3; multi-component recovery to r = 5. Corrections to the attention-chain rate, SwiGLU composition, the LayerNorm bracket, and the curvature-rate chain
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2606.05957 [cs.LG]
(or arXiv:2606.05957v2 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 320 cuvinte
Subiecte conexe
Deconfusing Corrigibility
AI Alignment Forum · 17 oct. 2026
Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization
arXiv · 21 iul. 2026
"I am an AI Safety Researcher"
LessWrong · 23 sept. 2026
Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm
arXiv · 16 iun. 2026