analiza aiqbench
This arXiv paper proposes EVA-Cap, which trains omni language models to caption video and audio by breaking captions into atomic audiovisual events and optimizing for event completeness, attribution, and synchronization. The reported gains (+9.0% on ChronusAV for event-centric evaluation, +3.3% on WorldSense) come from the authors' own benchmarks and a self-curated dataset, so they need independent replication. Still, the approach of using structured, event-level reward targets with GRPO is a useful idea for anyone working on multimodal alignment.
EVA-Cap: Optimizing Audiovisual Video Captioning via Event-Centric Alignment
arXiv:2610.22186v1 Announce Type: cross Abstract: Recent advances in Omni Language Models (OLMs) provide a unified solution for audiovisual video captioning. Nevertheless, existing approaches predominantly treat captioning as an unstructured text generation process, neglecting the inherent semantic structure of videos and leading to suboptimal fine-grained audiovisual alignment. To bridge this gap, we propose EVA-Cap, a novel framework that decomposes captions into atomic audiovisual events and transforms unstructured text supervision into event-centric alignment. We train EVA-Cap via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) on a self-curated dataset, EVA-Data, which is built with event-centric quality control. During the GRPO phase, the model is explicitly optimized against three event-centric alignment targets: global event completeness, intra-event attribution, and inter-event synchronization. Extensive experiments demonstrate that EVA-Cap significantly outperforms existing methods in event-centric evaluation (e.g., +9.0% average gain on ChronusAV), while achieving competitive performance on general audiovisual captioning benchmarks (e.g., +3.3% on WorldSense).
Abstract:Recent advances in Omni Language Models (OLMs) provide a unified solution for audiovisual video captioning. Nevertheless, existing approaches predominantly treat captioning as an unstructured text generation process, neglecting the inherent semantic structure of videos and leading to suboptimal fine-grained audiovisual alignment. To bridge this gap, we propose EVA-Cap, a novel framework that decomposes captions into atomic audiovisual events and transforms unstructured text supervision into event-centric alignment. We train EVA-Cap via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) on a self-curated dataset, EVA-Data, which is built with event-centric quality control. During the GRPO phase, the model is explicitly optimized against three event-centric alignment targets: global event completeness, intra-event attribution, and inter-event synchronization. Extensive experiments demonstrate that EVA-Cap significantly outperforms existing methods in event-centric evaluation (e.g., +9.0% average gain on ChronusAV), while achieving competitive performance on general audiovisual captioning benchmarks (e.g., +3.3% on WorldSense).
Accepted at NeurIPS 2026. Code will be released at this https URL
Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
arXiv:2610.22186 [cs.CV]
(or arXiv:2610.22186v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2610.22186 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 257 cuvinte
Subiecte conexe
WALL-WM: Carving World Action Modeling at the Event Joints
arXiv · 9 sept. 2026
EventLens: Event-Structure Reinforcement Learning for Video Understanding
arXiv · 15 oct. 2026
EM^2Mem: Event-Centric Multimodal Memory for Large Language Models
arXiv · 2 sept. 2026
Vertical Advantage: Transforming Industries with Lakebase and Agentic AI
Databricks · 27 aug. 2026