aiqbench analysis
This research identifies a critical failure mode in linear probing where frozen pretrained features overfit noisy labels through specific spectral patterns, rather than just random noise. The study demonstrates that restricting the readout to intermediate principal component ranks can significantly recover clean accuracy compared to using the full feature space or standard validation-based selection. Ultimately, this work reframes PCA rank sweeping as a diagnostic tool for detecting structured overfitting in downstream tasks rather than merely a performance optimization technique.
Spectral Overfitting in Noisy Linear Probing of Pretrained Representations
arXiv:2603.02293v3 Announce Type: replace-cross Abstract: Frozen pretrained features are often treated as a safe interface for downstream learning: only a small linear readout is trained, while the backbone is fixed. We show that this readout can still overfit noisy labels in a structured way. A label-blind PCA rank sweep reveals a sharp spectral pattern: under label noise, exposing all pretrained directions can hurt clean accuracy, and intermediate ranks often recover much of the lost performance. Rank-matched random projections help less, and measured between-class signal is strongly concentrated in leading PCs. The pattern appears across three ImageNet-pretrained backbones on CIFAR-10, with gains up to $36.0\pm0.8$ points over the default full-rank probe at 40\% noise. Tuned full-rank probes outperform validation-selected PCA probes, so we present the sweep as a diagnostic of spectral overfitting rather than a competitive noisy-label method.
Abstract:Frozen pretrained features are often treated as a safe interface for downstream learning: only a small linear readout is trained, while the backbone is fixed. We show that this readout can still overfit noisy labels in a structured way. A label-blind PCA rank sweep reveals a sharp spectral pattern: under label noise, exposing all pretrained directions can hurt clean accuracy, and intermediate ranks often recover much of the lost performance. Rank-matched random projections help less, and measured between-class signal is strongly concentrated in leading PCs. The pattern appears across three ImageNet-pretrained backbones on CIFAR-10, with gains up to $36.0\pm0.8$ points over the default full-rank probe at 40\% noise. Tuned full-rank probes outperform validation-selected PCA probes, so we present the sweep as a diagnostic of spectral overfitting rather than a competitive noisy-label method.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2603.02293 [cs.LG]
(or arXiv:2603.02293v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2603.02293 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 227 words
Related coverage
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
Is a Linear Probe Evidence of a Linear Representation?
arXiv · Oct 19, 2026
Differential privacy representation geometry for medical image analysis
arXiv · Sep 16, 2026
Turning spectra into images improves plant trait retrieval with 2D-CNNs
arXiv · Aug 19, 2026