Multimodal Dataset Normalization and Perceptual Validation for Music-Taste Correspondences

Abstract:Music and food traditions are both intangible cultural heritage, and the links between them, how a sound can make a taste seem sweeter or more bitter, are increasingly used in museum, exhibition and gastronomic-tourism settings. Modelling those links computationally runs into a data bottleneck familiar across cultural heritage computing: expert annotation is slow and costly, so the annotated collections that result are small. The usual remedy is to enlarge a collection automatically, labelling it with a model trained on the small annotated one. Such synthetic labels are rarely checked, either against the original annotations or against people. We provide both checks. Experiment 1 asks whether the audio-flavour patterns found in an experimental soundtrack collection (257 tracks annotated by listeners) survive when the collection is scaled to 49,300 30-second segments from the Free Music Archive (FMA) labelled by a fine-tuned Audio Spectrogram Transformer. Experiment 2 asks whether flavour profiles computed from food chemistry, for 20 dishes drawn largely from Italian culinary tradition, match what listeners actually hear (49 participants, online). Feature-flavour patterns carry over for every taste dimension (rho=0.38-0.72, all p<0.001), and sweetness still carries over when every spectral feature is removed, so the agreement is not an artefact of how the labelling model represents audio. Listener ratings match the computed profiles far beyond chance (permutation p<0.001; Mantel r=0.45; Procrustes m^2=0.49), and the result holds when participants reporting hearing or taste impairments are excluded. We release the harmonized datasets and all code.

Sound (cs.SD); Machine Learning (cs.LG); Multimedia (cs.MM); Audio and Speech Processing (eess.AS)

arXiv:2604.10632 [cs.SD]

(or arXiv:2604.10632v2 [cs.SD] for this version)

https://doi.org/10.48550/arXiv.2604.10632 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.