aiqbench analysis
KREPES offers a unified analytical framework that interprets self-supervised learning models like SimCLR and BYOL by linking neural tangent kernels to the Representer Theorem. This approach enables direct auditing of latent spaces to detect algorithmic biases and quantify transparency without requiring supervision, while Nyström approximation ensures scalability to datasets exceeding one million samples. The method provides concrete metrics for influence scoring and feature alignment, addressing the critical need for explainability in large-scale unsupervised training.
Interpretable Self-Supervised Learning via Representer Landmarks and Nystr\"om Approximation
arXiv:2509.24467v4 Announce Type: replace Abstract: Self-supervised learning (SSL) learns representations from massive unlabeled data, yet the resulting models typically operate as black boxes, necessitating domain-specific explanations. We introduce KREPES, a unified framework to analytically interpret the learned representations of SSL objectives, including SimCLR, BYOL, and VICReg. By bridging empirical neural tangent kernel approximations of neural networks with the Representer Theorem for kernels, we express the learned latent space directly via "Representer Landmarks", which are the representations of influential unlabeled training examples. We introduce novel metrics, "Sample-Specific Influence Score", "Concept-Conditioned Influence Score" and "Feature Alignment Gap", to quantify the transparency of the learned representations. KREPES enables direct audit of the latent space without supervision, for example, revealing an algorithmic bias in the Adult-1M dataset where SSL uses demographic proxies for income. Finally, to ensure scalability to benchmarks with 1M+ samples (ImageNet-1K, Adult-1M), KREPES introduces a novel Nystr\"om approximation-based analytical inference framework for SSL objectives.
Abstract:Self-supervised learning (SSL) learns representations from massive unlabeled data, yet the resulting models typically operate as black boxes, necessitating domain-specific explanations. We introduce KREPES, a unified framework to analytically interpret the learned representations of SSL objectives, including SimCLR, BYOL, and VICReg. By bridging empirical neural tangent kernel approximations of neural networks with the Representer Theorem for kernels, we express the learned latent space directly via "Representer Landmarks", which are the representations of influential unlabeled training examples. We introduce novel metrics, "Sample-Specific Influence Score", "Concept-Conditioned Influence Score" and "Feature Alignment Gap", to quantify the transparency of the learned representations. KREPES enables direct audit of the latent space without supervision, for example, revealing an algorithmic bias in the Adult-1M dataset where SSL uses demographic proxies for income. Finally, to ensure scalability to benchmarks with 1M+ samples (ImageNet-1K, Adult-1M), KREPES introduces a novel Nyström approximation-based analytical inference framework for SSL objectives.
24 pages, 10 figures. Accepted to the 43rd International Conference on Machine Learning (ICML 2026)
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2509.24467 [cs.LG]
(or arXiv:2509.24467v4 [cs.LG] for this version)
Full text under the publisher's open licence · 177 words
Related coverage
Certified Approximation for Interpretable Representer Landmarks
arXiv · Oct 1, 2026
How do neural networks regress cubic polynomials? Apparently, they use a trick invented in Milan 500 years ago
LessWrong · Jul 22, 2026
Models Can't Remember Their Training. Neither Can You.
LessWrong · Jul 19, 2026
Optimization Encoders: Rethinking Second-Order Meta-Learning for Neural Fields
arXiv · Oct 7, 2026