analiza aiqbench
This research introduces FuSiLi, a method for multimodal contrastive learning that derives fine-grained local alignments between audio frames and sheet music images using only coarse global supervision. By leveraging Sinkhorn-based soft alignment on pretrained CLIP/CLAP encoders, the approach effectively bridges the gap where direct local correspondence data is scarce while maintaining strong retrieval performance. The technique advances machine learning in multimodal domains by demonstrating that high-fidelity temporal-spatial mapping can be learned indirectly from limited labels.
Local Multimodal Music Alignment from Global Supervision
arXiv:2607.10023v2 Announce Type: replace-cross Abstract: Understanding music requires understanding localized relationships across data modalities, e.g., how time in performance audio maps onto position in a score image. Yet supervision for such local correspondences is difficult to obtain-in practice, we often only have access to coarser global supervision like paired segments of audio and images. To address this gap, we propose FuSiLi (Fused Sinkhorn-Localized Similarity), a similarity score for multimodal contrastive learning operating directly on local image patch and audio frame features via Sinkhorn-based soft alignment. We show that FuSiLi (i) effectively learns local relationships, (ii) requires only global supervision, and (iii) retains the global alignment capabilities of conventional contrastive approaches. We fine-tune pretrained CLIP and CLAP encoders on pairs of raw sheet music images and audio using a hybrid contrastive objective combining FuSiLi with conventional global similarity. We evaluate on cross-modal retrieval and frame-level alignment tasks against a range of global and local baselines, showing that our approach outperforms them on local alignment while remaining competitive on retrieval.
Abstract:Understanding music requires understanding localized relationships across data modalities, e.g., how time in performance audio maps onto position in a score image. Yet supervision for such local correspondences is difficult to obtain-in practice, we often only have access to coarser global supervision like paired segments of audio and images. To address this gap, we propose FuSiLi (Fused Sinkhorn-Localized Similarity), a similarity score for multimodal contrastive learning operating directly on local image patch and audio frame features via Sinkhorn-based soft alignment. We show that FuSiLi (i) effectively learns local relationships, (ii) requires only global supervision, and (iii) retains the global alignment capabilities of conventional contrastive approaches. We fine-tune pretrained CLIP and CLAP encoders on pairs of raw sheet music images and audio using a hybrid contrastive objective combining FuSiLi with conventional global similarity. We evaluate on cross-modal retrieval and frame-level alignment tasks against a range of global and local baselines, showing that our approach outperforms them on local alignment while remaining competitive on retrieval.
Sound (cs.SD); Machine Learning (cs.LG); Multimedia (cs.MM)
arXiv:2607.10023 [cs.SD]
(or arXiv:2607.10023v2 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2607.10023 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 259 cuvinte
Subiecte conexe
FedADB: Class Anchor-Driven Dual-Branch Federated Learning for Mitigating Forgetting
arXiv · 19 aug. 2026
Unlocking Spatial Grounding in Large Audio-Visual Retrieval models
arXiv · 29 iul. 2026
MarsRecon: Self-Supervised and Multimodal Surface Representations for Mars
arXiv · 22 sept. 2026
NVIDIA GTC Taipei at COMPUTEX: Live Updates on What’s Next in AI
NVIDIA · 21 mai 2026