How AI Experiences Art: Emergent Aesthetic Structure in a Self-Supervised Multimodal Embedding Space

Abstract:Aesthetics are an important part of the symbolism of artistic works. Although subjective, humans categorize art based on the emotion evoked regardless of modality. What remains under-explored is how AI models form their own aesthetic categorization of human-produced media without explicit labels or cross-modal supervision. We present a self-supervised framework that projects four modalities (text, audio, image and video) into a shared 256-dimensional embedding space and applies iterative clustering to discover aesthetic structure. We discuss the divergence between AI-generated cluster assignments and human affective register labels on a weakly supervised multimodal dataset. This work has applications in understanding how AI structures cross-modal similarity, organizing heterogeneous media collections for Retrieval-Augmented Generation (RAG), and automated data labeling.

Accepted at ICMI Companion '26 (Companion Publication of the 28th ACM International Conference on Multimodal Interaction), October 5-9, 2026, Napoli, Italy. 4 figures, 2 tables

Multimedia (cs.MM); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)

I.2.6; I.5.3; H.5.1

arXiv:2608.27121 [cs.MM]

(or arXiv:2608.27121v1 [cs.MM] for this version)

https://doi.org/10.48550/arXiv.2608.27121 arXiv-issued DOI via DataCite (pending registration)

https://doi.org/10.1145/3776591.3837060 DOI(s) linking to related resources