Multi-Modal Scene Graph with Kolmogorov-Arnold Experts for Audio-Visual Question Answering

arXiv:2511.23304v2 Announce Type: replace Abstract: In this paper, we propose a novel Multi-Modal Scene Graph with Kolmogorov-Arnold Expert Network for Audio-Visual Question Answering (SHRIKE). The task aims to mimic human reasoning by extracting and fusing information from audio-visual scenes, with the main challenge being the identification of question-relevant cues from complex audio-visual content. Existing methods fail to capture the structural information within videos and suffer from insufficient fine-grained modeling of multi-modal features. To address these issues, we are the first to introduce a new multi-modal scene graph that explicitly models objects and their relationships as a visually grounded, structured representation of the audio-visual scene, yielding 461,292 relation triplets over 9,288 musical performance videos. Furthermore, we design a Kolmogorov-Arnold Network (KAN)-based Mixture of Experts (MoE) to enhance the expressive power of the temporal integration stage. This enables more fine-grained modeling of cross-modal interactions within the question-aware fused audio-visual representation, capturing richer and more nuanced patterns and improving temporal reasoning performance. We evaluate the model on the established MUSIC-AVQA and MUSIC-AVQA v2 benchmarks, where it achieves state-of-the-art performance, reaching 78.14% average accuracy on MUSIC-AVQA, surpassing the previous best method QA-TIGER and ranking first under all four configurations of MUSIC-AVQA v2.0. Code and model checkpoints will be publicly released at https://github.com/feel12348/SHRIKE.

Abstract:In this paper, we propose a novel Multi-Modal Scene Graph with Kolmogorov-Arnold Expert Network for Audio-Visual Question Answering (SHRIKE). The task aims to mimic human reasoning by extracting and fusing information from audio-visual scenes, with the main challenge being the identification of question-relevant cues from complex audio-visual content. Existing methods fail to capture the structural information within videos and suffer from insufficient fine-grained modeling of multi-modal features. To address these issues, we are the first to introduce a new multi-modal scene graph that explicitly models objects and their relationships as a visually grounded, structured representation of the audio-visual scene, yielding 461,292 relation triplets over 9,288 musical performance videos. Furthermore, we design a Kolmogorov-Arnold Network (KAN)-based Mixture of Experts (MoE) to enhance the expressive power of the temporal integration stage. This enables more fine-grained modeling of cross-modal interactions within the question-aware fused audio-visual representation, capturing richer and more nuanced patterns and improving temporal reasoning performance. We evaluate the model on the established MUSIC-AVQA and MUSIC-AVQA v2 benchmarks, where it achieves state-of-the-art performance, reaching 78.14% average accuracy on MUSIC-AVQA, surpassing the previous best method QA-TIGER and ranking first under all four configurations of MUSIC-AVQA v2.0. Code and model checkpoints will be publicly released at this https URL.

Artificial Intelligence (cs.AI)

arXiv:2511.23304 [cs.AI]

(or arXiv:2511.23304v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2511.23304 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

Multi-Modal Scene Graph with Kolmogorov-Arnold Experts for Audio-Visual Question Answering

Subiecte conexe