Structured-Noise Masked Modeling for Video, Audio and Beyond

Abstract:Masked modeling has emerged as a robust self-supervised learning framework. However, most methods rely on random masking, which disregards the structural properties of different data modalities. To align with the spatiotemporal and spectral characteristics of video and audio data, we introduce a structured noise-based masking approach. By filtering white noise into different color noise distributions, we generate structured masks that capture modality-specific patterns without requiring handcrafted heuristics or access to the data. Our approach enhances masked video and audio modeling frameworks without any additional computational cost. Experiments show that structured noise masking consistently outperforms random masking, underscoring the value of modality-aware masking strategies for representation learning.

ECCV 2026 (Oral). 37 pages, including supplementary material. Project page: this https URL

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Sound (cs.SD)

arXiv:2503.16311 [cs.LG]

(or arXiv:2503.16311v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2503.16311 Focus to learn more arXiv-issued DOI via DataCite

Computer Vision - ECCV 2026, Lecture Notes in Computer Science, vol. 17018, pp. 298-316, Springer, 2026

https://doi.org/10.1007/978-3-032-37517-9_17 Focus to learn more DOI(s) linking to related resources

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.