aiqbench analysis
This research introduces the TDGP model, which utilizes Transformer architectures to fuse visual and acoustic data for robust audio-visual navigation in environments with incomplete perception. By integrating dynamic graph planning with collision penalties, the system enables agents to autonomously replan routes when visual cues are misleading, significantly improving safety and generalization without relying on physical impacts. The approach demonstrates superior performance over baseline methods on standard indoor navigation datasets, highlighting advances in multimodal sensor fusion for autonomous agents.
Transformer-Based Token Fusion and Dynamic Graph Planning for Audio-Visual Navigation
arXiv:2609.17421v1 Announce Type: new Abstract: Audio-Visual Navigation (AVN) requires an agent to localize and navigate toward a continuously vocalizing target relying solely on visual observations and acoustic cues. Currently, systems lack the ability to adaptively correct and replan when faced with incomplete or misleading visual perception. Furthermore, relying on physical collisions to compensate for missing visual information results in inefficient and unsafe navigation, whereas existing methods are overly dependent on passive visual perception. To address these issues, we propose the Transformer-based Token Fusion and Dynamic Graph Planning (TDGP) model, which incorporates high-level perception layers and leverages the Transformer model to fuse multimodal cues for precise local planning. Next, a low-level planning layer is designed that uses physical collision penalties to remove edges that collide with the map in real time and apply corresponding penalties, forcing the agent to automatically re-plan to compensate for the lack of visual information. Experiments show that our TDGP model outperforms baseline models on the Replica and Matterport3D (MP3D) datasets, and that the model's sound enhancement strategy significantly improves generalization in unheard acoustic scenarios.
Abstract:Audio-Visual Navigation (AVN) requires an agent to localize and navigate toward a continuously vocalizing target relying solely on visual observations and acoustic cues. Currently, systems lack the ability to adaptively correct and replan when faced with incomplete or misleading visual perception. Furthermore, relying on physical collisions to compensate for missing visual information results in inefficient and unsafe navigation, whereas existing methods are overly dependent on passive visual perception. To address these issues, we propose the Transformer-based Token Fusion and Dynamic Graph Planning (TDGP) model, which incorporates high-level perception layers and leverages the Transformer model to fuse multimodal cues for precise local planning. Next, a low-level planning layer is designed that uses physical collision penalties to remove edges that collide with the map in real time and apply corresponding penalties, forcing the agent to automatically re-plan to compensate for the lack of visual information. Experiments show that our TDGP model outperforms baseline models on the Replica and Matterport3D (MP3D) datasets, and that the model's sound enhancement strategy significantly improves generalization in unheard acoustic scenarios.
Main paper (6 pages). Accepted for publication by IEEE International Conference on Systems, Man, and Cybernetics 2026 (IEEE SMC 2026)
Artificial Intelligence (cs.AI); Signal Processing (eess.SP)
arXiv:2609.17421 [cs.AI]
(or arXiv:2609.17421v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.17421 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 290 words
Related coverage
News
Quantum Computing Report · Jul 25, 2026
Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine
NVIDIA · Sep 14, 2026
NVIDIA GTC Taipei at COMPUTEX: Live Updates on What’s Next in AI
NVIDIA · May 21, 2026
Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
Hugging Face · Aug 18, 2026