analiza aiqbench
This research addresses representation collapse in diffusion models by aligning internal features across denoising layers, specifically optimizing them for video-background music generation. The proposed U-SRA architecture successfully improves multimodal alignment and symbolic music quality on benchmark datasets, offering a concrete architectural improvement for generative audio tasks.
Self-Representation Alignment in Diffusion Models for Video Background Music Generation
arXiv:2610.15575v1 Announce Type: new Abstract: Diffusion models have been widely used for various generative tasks. However, recent studies have demonstrated that the internal representations learned in the diffusion models' denoising process are insufficiently meaningful, which can hinder the generation performance. To address this, we propose a novel diffusion U-Net architecture enhanced by explicitly aligning the weaker representation from the earlier layer conditioned on higher noise with the better representation from the later layer conditioned on lower noise during training, which we use for the challenging multimodal task of video background music generation. Our diffusion U-Net Self-Representation Alignment (U-SRA) employs an alignment loss to effectively guide our diffusion model to learn good internal representations, thereby enabling it to produce higher-quality symbolic music conditioned on video features. We employ a cross-attention mechanism to synchronize music with video segments. Experiments on a benchmark video-music dataset demonstrate that the proposed U-SRA approach for video background music generation surpasses the state-of-the-art across key evaluation metrics, including video-music alignment and overall music quality.
Abstract:Diffusion models have been widely used for various generative tasks. However, recent studies have demonstrated that the internal representations learned in the diffusion models' denoising process are insufficiently meaningful, which can hinder the generation performance. To address this, we propose a novel diffusion U-Net architecture enhanced by explicitly aligning the weaker representation from the earlier layer conditioned on higher noise with the better representation from the later layer conditioned on lower noise during training, which we use for the challenging multimodal task of video background music generation. Our diffusion U-Net Self-Representation Alignment (U-SRA) employs an alignment loss to effectively guide our diffusion model to learn good internal representations, thereby enabling it to produce higher-quality symbolic music conditioned on video features. We employ a cross-attention mechanism to synchronize music with video segments. Experiments on a benchmark video-music dataset demonstrate that the proposed U-SRA approach for video background music generation surpasses the state-of-the-art across key evaluation metrics, including video-music alignment and overall music quality.
Artificial Intelligence (cs.AI)
arXiv:2610.15575 [cs.AI]
(or arXiv:2610.15575v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.15575 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 256 cuvinte
Subiecte conexe
Links #5: 2026/07
LessWrong · 31 iul. 2026
Triton-Based Optimization of Video Sparse Attention on ROCm
AMD · 13 iul. 2026
VIBE: Video Instruction-aligned Background music gEneration
arXiv · 1 sept. 2026
Dyna-2: General Purpose Physical Agents with Hour-Long Autonomy — Jason Ma, Dyna Robotics
Latent Space · 13 oct. 2026