aiqbench analysis
Audio-JEPA introduces a self-supervised learning framework that leverages the Joint-Embedding Predictive Architecture to predict latent spectrogram representations rather than reconstructing raw audio, significantly reducing data requirements compared to standard models like wav2vec 2.0. By utilizing a Vision Transformer backbone for mel-spectrogram patches, this approach achieves comparable performance across speech, music, and environmental sound tasks with minimal hyper-parameter tuning. The method represents a notable efficiency gain in audio representation learning by decoupling the need for massive labeled datasets or raw signal reconstruction.
Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning
arXiv:2507.02915v2 Announce Type: replace-cross Abstract: Building on the Joint-Embedding Predictive Architecture (JEPA) paradigm, a recent self-supervised learning framework that predicts latent representations of masked regions in high-level feature spaces, we propose Audio-JEPA (Audio Joint-Embedding Predictive Architecture), tailored specifically for audio data. Audio-JEPA uses a simple Vision Transformer backbone to predict latent representations of masked spectrogram patches rather than reconstructing raw audio. We pre-train on unlabeled AudioSet clips (10s, 32kHz) with random patch masking on mel-spectrograms. We evaluate on the X-ARES suite covering speech, music, and environmental sound tasks. Although our implementation is a straightforward translation of the original model to audio, the results still show comparable performance to wav2vec 2.0 and data2vec while using less than one-fifth of their training data and with no hyper-parameter tuning. All code and pretrained checkpoints are available on GitHub.
Abstract:Building on the Joint-Embedding Predictive Architecture (JEPA) paradigm, a recent self-supervised learning framework that predicts latent representations of masked regions in high-level feature spaces, we propose Audio-JEPA (Audio Joint-Embedding Predictive Architecture), tailored specifically for audio data. Audio-JEPA uses a simple Vision Transformer backbone to predict latent representations of masked spectrogram patches rather than reconstructing raw audio. We pre-train on unlabeled AudioSet clips (10s, 32kHz) with random patch masking on mel-spectrograms. We evaluate on the X-ARES suite covering speech, music, and environmental sound tasks. Although our implementation is a straightforward translation of the original model to audio, the results still show comparable performance to wav2vec 2.0 and data2vec while using less than one-fifth of their training data and with no hyper-parameter tuning. All code and pretrained checkpoints are available on GitHub.
Sound (cs.SD); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS); Signal Processing (eess.SP)
arXiv:2507.02915 [cs.SD]
(or arXiv:2507.02915v2 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2507.02915 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 235 words
Related coverage
Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning
arXiv · Jun 6, 2026
CoJEPA: Combining Contrastive Learning and JEPA for Global-Local Music Representations
arXiv · Sep 1, 2026
The JEPA Paradox in Language: The Geometry of Linguistic Alternatives
arXiv · Jul 28, 2026
Mask-Aware Execution for Efficient JEPA Training
arXiv · Sep 22, 2026