aiqbench analysis
This research identifies a critical flaw in standard supervised learning where minimizing task loss ignores representation drift along nuisance directions, creating what the authors term a "geometric blind spot." The study demonstrates through theoretical proofs on linear-Gaussian models and empirical results on Vision Transformers that adversarial training can inadvertently worsen clean class geometry while reducing Jacobian magnitude. To address this, the paper proposes isotropic encoder matching as a robust design rule for maintaining stable representations when nuisance axes are unknown.
Supervised Learning Has a Geometric Blind Spot
arXiv:2604.21395v3 Announce Type: replace Abstract: Ordinary supervised training minimises the task loss and then stops. It never pays for how far the representation moves when the input is nudged along directions that helped fit training labels---including directions that are nuisance at deployment. We call that leftover sensitivity the geometric blind spot of empirical risk minimisation. In a Gaussian linear model where the nuisance enters the label conditional and the decoder has finite Lipschitz constant, population MSE forces a floor on linearised representation drift. The same distinction predicts a failure mode of adversarial training: Jacobian magnitude can fall while clean class geometry worsens. We track that dissociation with a class-layout score and study isotropic encoder matching---penalising the squared distance between phi(x) and phi(x+delta) for Gaussian delta under a task-loss cap---when nuisance axes are unknown. On a Vision Transformer trained from scratch on CIFAR-10, projected gradient descent attains the smallest Jacobian Frobenius yet the worst clean layout score (1.353+/-0.020 over three seeds), above task-only training (1.093); isotropic matching attains the best (0.904). The drift floor is proved for the linear-Gaussian case; deep nets and cross-task orderings are protocol empirics. Design rule: report class-layout geometry beside the task score; prefer isotropic encoder matching when axes are unknown.
Abstract:Ordinary supervised training minimises the task loss and then stops. It never pays for how far the representation moves when the input is nudged along directions that helped fit training labels---including directions that are nuisance at deployment. We call that leftover sensitivity the geometric blind spot of empirical risk minimisation. In a Gaussian linear model where the nuisance enters the label conditional and the decoder has finite Lipschitz constant, population MSE forces a floor on linearised representation drift. The same distinction predicts a failure mode of adversarial training: Jacobian magnitude can fall while clean class geometry worsens. We track that dissociation with a class-layout score and study isotropic encoder matching---penalising the squared distance between phi(x) and phi(x+delta) for Gaussian delta under a task-loss cap---when nuisance axes are unknown. On a Vision Transformer trained from scratch on CIFAR-10, projected gradient descent attains the smallest Jacobian Frobenius yet the worst clean layout score (1.353+/-0.020 over three seeds), above task-only training (1.093); isotropic matching attains the best (0.904). The drift floor is proved for the linear-Gaussian case; deep nets and cross-task orderings are protocol empirics. Design rule: report class-layout geometry beside the task score; prefer isotropic encoder matching when axes are unknown.
35 pages. v2: JMLR-aligned revision of arXiv:2604.21395; Proposition 6 corrected to minimax (worst-case) anisotropy; title shortened to Supervised Learning Has a Geometric Blind Spot. Under submission at JMLR. Companion: arXiv:2605.22800
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2604.21395 [cs.LG]
(or arXiv:2604.21395v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2604.21395 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 330 words
Related coverage
CorrelationFlow: A Training-Free Geometric Approach for LiDAR Scene Flow Estimation
arXiv · Aug 3, 2026
Evaluating Risks in Weak-to-Strong Alignment: A Bias-Variance Perspective
arXiv · Jul 24, 2026
A Geometric Unification of Concept Learning with Concept Cones
arXiv · Jun 9, 2026
Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models
arXiv · Aug 5, 2026