analiza aiqbench
AV-JEPA introduces a streamlined architecture for audio-visual self-supervised learning by leveraging early-fusion Vision Transformers and modality dropout to achieve cross-modal alignment without complex decoders or contrastive negatives. The approach demonstrates competitive performance on standard benchmarks like VGGSound and AudioSet while enabling zero-shot retrieval, highlighting its potential efficiency in multimodal foundation models.
AV-JEPA: Extending LeJEPA to Audio-Visual Self-Supervised Learning
arXiv:2607.15295v1 Announce Type: cross Abstract: We present AV-JEPA, an elegant multimodal extension of LeJEPA to audio-visual self-supervised learning. Using an early-fusion Vision Transformer and modality dropout as masking, the model is trained to align the embeddings of global and per-modality local views, while the SIGReg objective encourages a theoretically optimal distribution. This achieves cross-modal alignment in the latent space, resulting in a remarkably clean architecture with no decoder, EMA teacher, complex multi-term losses, or contrastive negatives. The proposed AV-JEPA backbone delivers competitive classification performance on VGGSound (57.1% top-1) and AudioSet (32.7 mAP) and supports zero-shot audio-video retrieval out of the box.
Abstract:We present AV-JEPA, an elegant multimodal extension of LeJEPA to audio-visual self-supervised learning. Using an early-fusion Vision Transformer and modality dropout as masking, the model is trained to align the embeddings of global and per-modality local views, while the SIGReg objective encourages a theoretically optimal distribution. This achieves cross-modal alignment in the latent space, resulting in a remarkably clean architecture with no decoder, EMA teacher, complex multi-term losses, or contrastive negatives. The proposed AV-JEPA backbone delivers competitive classification performance on VGGSound (57.1% top-1) and AudioSet (32.7 mAP) and supports zero-shot audio-video retrieval out of the box.
Multimedia (cs.MM); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Sound (cs.SD)
arXiv:2607.15295 [cs.MM]
(or arXiv:2607.15295v1 [cs.MM] for this version)
https://doi.org/10.48550/arXiv.2607.15295 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 195 cuvinte
Subiecte conexe
LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning
arXiv · 6 oct. 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · 25 sept. 2026
MR-JEPA: A General Purpose Video Foundation Model for Cardiac MRI
arXiv · 1 sept. 2026
Complete Guide to Thinking Machines Inkling
Analytics Vidhya · 20 iul. 2026