aiqbench analysis
This research introduces KARAMEL, a multimodal entity linking framework that integrates vision-language encoders with heterogeneous graph neural networks to leverage sociohistorical context. By moving beyond surface-level alignment, the system effectively disambiguates entities in cultural heritage datasets like WikiMuSA, demonstrating that structured domain knowledge significantly improves performance on implicit mentions. The work advances fundamental machine learning techniques for knowledge discovery within multimodal scientific settings.
KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphs
arXiv:2610.16502v1 Announce Type: cross Abstract: Discovering and disambiguating entities from multimodal and heterogeneous data is a key challenge in discovery science, where relevant knowledge is often implicit and distributed across modalities. We present KARAMEL, a framework for multimodal entity linking (MEL) - the task of aligning visual and textual mentions to entities in a knowledge base - in the cultural heritage domain. Unlike prior vision-language models that primarily rely on surface-level multimodal alignment, our approach incorporates sociohistorical and contextual knowledge through a knowledge-aware architecture that fuses a vision-language encoder with a heterogeneous graph neural network. This design enables contextual reasoning over complex relational structures and supports robust entity disambiguation under ambiguous or underspecified evidence. To benchmark multimodal discovery in this setting, we introduce WikiMuSA (Wikidata-based Multimodal Semantic data for Art), a new dataset that links artworks to Wikidata entities via images, textual descriptions, and structured contextual knowledge. Experimental results show that KARAMEL surpasses strong baselines, particularly on implicit or abstract mentions, while ablation studies highlight the critical role of domain-specific, structured context in learning expressive representations. While instantiated in the cultural heritage domain, KARAMEL addresses a general knowledge discovery problem and applies to other scientific settings involving multimodal observations and heterogeneous background knowledge. Both the dataset and source code are made publicly available.
Abstract:Discovering and disambiguating entities from multimodal and heterogeneous data is a key challenge in discovery science, where relevant knowledge is often implicit and distributed across modalities. We present KARAMEL, a framework for multimodal entity linking (MEL) - the task of aligning visual and textual mentions to entities in a knowledge base - in the cultural heritage domain. Unlike prior vision-language models that primarily rely on surface-level multimodal alignment, our approach incorporates sociohistorical and contextual knowledge through a knowledge-aware architecture that fuses a vision-language encoder with a heterogeneous graph neural network. This design enables contextual reasoning over complex relational structures and supports robust entity disambiguation under ambiguous or underspecified evidence. To benchmark multimodal discovery in this setting, we introduce WikiMuSA (Wikidata-based Multimodal Semantic data for Art), a new dataset that links artworks to Wikidata entities via images, textual descriptions, and structured contextual knowledge. Experimental results show that KARAMEL surpasses strong baselines, particularly on implicit or abstract mentions, while ablation studies highlight the critical role of domain-specific, structured context in learning expressive representations. While instantiated in the cultural heritage domain, KARAMEL addresses a general knowledge discovery problem and applies to other scientific settings involving multimodal observations and heterogeneous background knowledge. Both the dataset and source code are made publicly available.
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
arXiv:2610.16502 [cs.CV]
(or arXiv:2610.16502v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2610.16502 arXiv-issued DOI via DataCite (pending registration)
Scaringi, R., Sierra-Múnera, A., Vessio, G. et al. KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphs. Mach Learn 115, 243 (2026)
https://doi.org/10.1007/s10994-026-07172-1 DOI(s) linking to related resources
Full text under the publisher's open licence · 264 words
Related coverage
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026
Digital States 2026: AI Moves From Experiment to Infrastructure
govtech.com · Sep 24, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026
When Generated Images Look Right and Retrieve Wrong: Coverage-Guided Cross-Scale Re-Indexing for Knowledge-Faithful Generative Perception
arXiv · Aug 24, 2026