aiqbench analysis
This research addresses fine-grained entity misalignment in synthetic training data for zero-shot image captioning, proposing a framework to enforce explicit correspondence between detected objects and generated text descriptions. By integrating an adaptive weighting strategy that downgrades unreliable pairs, the method significantly improves supervision fidelity without requiring new annotated datasets. The approach offers a practical upgrade for existing pipelines aiming to enhance consistency across both in-domain and cross-domain benchmarks.
Entity-Faithful Repair of Synthetic Supervision for Zero-Shot Image Captioning
arXiv:2608.00994v1 Announce Type: cross Abstract: Zero-shot image captioning aims to generate image descriptions without annotated image-text pairs. Recent approaches exploit text-to-image models to synthesize training data from text-only corpora, but most focus on improving overall data quality. In contrast, we observe that synthetic image-text misalignment is often structured and fine-grained: pairs may remain globally plausible while containing missing entities or misgrounded attributes, thereby degrading supervision fidelity. As a result, methods based on global similarity for image rematching or regeneration may improve apparent plausibility, but cannot systematically repair entity-level misalignment. To address this issue, we propose ReCap, a plug-and-play framework that shifts synthetic data refinement from implicit global matching to explicit fine-grained realignment. Specifically, ReCap enforces entity-level correspondence by using detected image-supported entities to guide caption rewriting, yielding more faithful synthetic supervision. In addition, we introduce an adaptive dynamic weighted learning strategy to downweight unreliable synthetic pairs during training. As a general framework, ReCap can be integrated into existing synthetic-data pipelines. Extensive experiments show that ReCap consistently improves image-text consistency and achieves state-of-the-art performance on both in-domain and cross-domain zero-shot image captioning benchmarks.
Abstract:Zero-shot image captioning aims to generate image descriptions without annotated image-text pairs. Recent approaches exploit text-to-image models to synthesize training data from text-only corpora, but most focus on improving overall data quality. In contrast, we observe that synthetic image-text misalignment is often structured and fine-grained: pairs may remain globally plausible while containing missing entities or misgrounded attributes, thereby degrading supervision fidelity. As a result, methods based on global similarity for image rematching or regeneration may improve apparent plausibility, but cannot systematically repair entity-level misalignment. To address this issue, we propose ReCap, a plug-and-play framework that shifts synthetic data refinement from implicit global matching to explicit fine-grained realignment. Specifically, ReCap enforces entity-level correspondence by using detected image-supported entities to guide caption rewriting, yielding more faithful synthetic supervision. In addition, we introduce an adaptive dynamic weighted learning strategy to downweight unreliable synthetic pairs during training. As a general framework, ReCap can be integrated into existing synthetic-data pipelines. Extensive experiments show that ReCap consistently improves image-text consistency and achieves state-of-the-art performance on both in-domain and cross-domain zero-shot image captioning benchmarks.
Accepted to the 34th ACM International Conference on Multimedia (ACM MM 2026). 16 pages, 7 figures
Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
arXiv:2608.00994 [cs.CV]
(or arXiv:2608.00994v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2608.00994 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 294 words
Related coverage
Echoverse: Deep, evolving environments for computer-use agents
Microsoft Research · Jul 30, 2026
FigEx2: Visual-Conditioned Panel Detection and Captioning for Scientific Compound Figures
arXiv · Sep 9, 2026
SUFLECA: Scaling Up Feature Learning for CAD-to-image Alignment
arXiv · Aug 31, 2026
World Models and Yann LeCun’s Bet Against Large Language Models
Quantum Zeitgeist · Jul 17, 2026