An Empirical Study into Clustering of Unseen Datasets with Self-Supervised Encoders

arXiv:2406.02465v2 Announce Type: replace-cross Abstract: Can pretrained models generalize to new datasets without any retraining? We deploy pretrained image models on datasets they were not trained for, and investigate whether their embeddings form meaningful clusters. Our suite of benchmarking experiments uses encoders pretrained solely on ImageNet-1k with either supervised or self-supervised training techniques, deployed on image datasets that were not seen during training, and clustered with conventional clustering algorithms. This evaluation provides new insights into the embeddings of self-supervised models, which prioritize different features to supervised models. We find evidence that supervised encoders offer more utility than SSL encoders within the training domain, and vice-versa far outside of it. However, fine-tuning SSL encoders for ImageNet-1k classification results in the opposite behaviour, with better performance than supervised-only models on in-domain and decreased performance on far out of domain data - worse at far-OOD than either SSL-only or supervised-only models. Clustering provides a way to evaluate the utility of self-supervised learnt representations orthogonal to existing feature quality estimation methods. Additionally, we find the silhouette score when measured in a UMAP-reduced space is highly correlated with clustering performance, and can therefore be used as a proxy for clustering performance on data with no ground truth labels. Our code implementation is available at https://github.com/scottclowe/zs-ssl-clustering/.

Abstract:Can pretrained models generalize to new datasets without any retraining? We deploy pretrained image models on datasets they were not trained for, and investigate whether their embeddings form meaningful clusters. Our suite of benchmarking experiments uses encoders pretrained solely on ImageNet-1k with either supervised or self-supervised training techniques, deployed on image datasets that were not seen during training, and clustered with conventional clustering algorithms. This evaluation provides new insights into the embeddings of self-supervised models, which prioritize different features to supervised models. We find evidence that supervised encoders offer more utility than SSL encoders within the training domain, and vice-versa far outside of it. However, fine-tuning SSL encoders for ImageNet-1k classification results in the opposite behaviour, with better performance than supervised-only models on in-domain and decreased performance on far out of domain data - worse at far-OOD than either SSL-only or supervised-only models. Clustering provides a way to evaluate the utility of self-supervised learnt representations orthogonal to existing feature quality estimation methods. Additionally, we find the silhouette score when measured in a UMAP-reduced space is highly correlated with clustering performance, and can therefore be used as a proxy for clustering performance on data with no ground truth labels. Our code implementation is available at this https URL.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)

arXiv:2406.02465 [cs.LG]

(or arXiv:2406.02465v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2406.02465 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

An Empirical Study into Clustering of Unseen Datasets with Self-Supervised Encoders

Related coverage