analiza aiqbench
PRISM-Bench addresses a critical gap in text-to-audio-video generation by introducing the first diagnostic benchmark that rigorously evaluates the audio modality rather than treating it as secondary to visual quality. By factorizing evaluation along axes of audio type and source visibility, this tool exposes significant performance disparities between frontier and open-source models, particularly in complex grounding tasks like synchronized music generation. This work is essential for researchers aiming to improve the fidelity and controllability of multimodal generative systems.
PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation
arXiv:2609.04867v1 Announce Type: cross Abstract: Text-to-audio-video (T2AV) generation has advanced rapidly, but its evaluation still underestimates the audio modality. Existing benchmarks either treat audio as an auxiliary component of video quality or assess it in isolation from audiovisual grounding, making it difficult to diagnose where current systems truly succeed or fail in audio generation. We present PRISM-Bench, the first audio-centric diagnostic benchmark for T2AV generation. Built from a rigorously curated dataset of 900 human-verified samples, PRISM-Bench factorizes audio evaluation along two orthogonal axes: audio type (Speech, Music, and Sound) and sound-source visibility (On-screen vs. Off-screen). It evaluates generated content across four perceptual dimensions (Audio-Visual Coherence, Audio Quality, Audio Expressiveness, and Prompt Following) with 35 fine-grained criteria. To ensure reliable assessment, we adopt an enhanced MLLM-as-a-Judge protocol based on blind, side-by-side comparison against ground-truth references, demonstrating strong alignment (over 70% mean agreement) with human raters. Our evaluation of recent T2AV systems highlights a significant performance gap between frontier and open-source models. Furthermore, we demonstrate that current generation paradigms overfit to perceptual fidelity while struggling with complex grounding and control tasks, particularly in generating music and synchronized On-screen audio.
Abstract:Text-to-audio-video (T2AV) generation has advanced rapidly, but its evaluation still underestimates the audio modality. Existing benchmarks either treat audio as an auxiliary component of video quality or assess it in isolation from audiovisual grounding, making it difficult to diagnose where current systems truly succeed or fail in audio generation. We present PRISM-Bench, the first audio-centric diagnostic benchmark for T2AV generation. Built from a rigorously curated dataset of 900 human-verified samples, PRISM-Bench factorizes audio evaluation along two orthogonal axes: audio type (Speech, Music, and Sound) and sound-source visibility (On-screen vs. Off-screen). It evaluates generated content across four perceptual dimensions (Audio-Visual Coherence, Audio Quality, Audio Expressiveness, and Prompt Following) with 35 fine-grained criteria. To ensure reliable assessment, we adopt an enhanced MLLM-as-a-Judge protocol based on blind, side-by-side comparison against ground-truth references, demonstrating strong alignment (over 70% mean agreement) with human raters. Our evaluation of recent T2AV systems highlights a significant performance gap between frontier and open-source models. Furthermore, we demonstrate that current generation paradigms overfit to perceptual fidelity while struggling with complex grounding and control tasks, particularly in generating music and synchronized On-screen audio.
19 pages, 10 figures, 4 tables. Accepted at ACM Multimedia 2026 (MM '26). This arXiv version includes supplementary appendices not included in the conference proceedings version
Multimedia (cs.MM); Artificial Intelligence (cs.AI); Sound (cs.SD)
arXiv:2609.04867 [cs.MM]
(or arXiv:2609.04867v1 [cs.MM] for this version)
https://doi.org/10.48550/arXiv.2609.04867 arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.1145/3767308.3836100 DOI(s) linking to related resources
Text integral sub licența deschisă a publicației · 236 cuvinte
Subiecte conexe
MUUNRiver-Bench: Diagnosing Relation-Dependent Music Retrieval with Multimodal Instructions
arXiv · 16 sept. 2026
MIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intent
arXiv · 8 oct. 2026
InfoVis: Unified Slide Image Generation and Editing
arXiv · 15 oct. 2026
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026