aiqbench analysis
VisExMEM introduces a novel framework for evaluating image-text models by decomposing captions into atomic claims and verifying them against localized visual evidence, addressing the opacity of global scoring metrics. By training Qwen3.5-based models on this approach, the authors demonstrate significant improvements in descriptive accuracy over existing open-source evaluators while maintaining competitiveness with top-tier proprietary solutions. This work advances the field of explainable AI by providing verifiable rationales for multimodal model outputs without requiring external reference data.
VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metric
arXiv:2610.16441v1 Announce Type: cross Abstract: Current image-text evaluation often relies on a global score, which gives limited insight into which semantic details are supported by the image. Explainable metrics provide rationales, but the visual evidence supporting those rationales can still be difficult to verify. We introduce VisExMEM, a reference-free evaluator that decomposes captions into atomic claims, localizes visual evidence, verifies each claim, and averages claim-level support into a global score. Explanations use the same evidence and decisions. We train Qwen3.5-based 4B and 9B models on 100K samples with multi-task SFT, counterfactual refinement, and evidence-aware GDPO. Across eight benchmarks, VisExMEM-9B achieves the highest descriptive composite among open-source evaluators in our comparison, exceeding VQAScore by 11.7 points while remaining competitive with GPT-5.6 Sol. We also introduce AGITA-Bench, containing 600 human-annotated image-text pairs and 8.7K atomic claims for global and claim-level evaluation. For untuned Qwen3.5 and GPT-5.6 Sol controls, decomposed prompting raises the same composite by 3.6-5.1 points relative to holistic prompting. Human evaluation shows that VisExMEM explanations outperform prior explainable metrics and approach GPT-5.6 Sol.
Abstract:Current image-text evaluation often relies on a global score, which gives limited insight into which semantic details are supported by the image. Explainable metrics provide rationales, but the visual evidence supporting those rationales can still be difficult to verify. We introduce VisExMEM, a reference-free evaluator that decomposes captions into atomic claims, localizes visual evidence, verifies each claim, and averages claim-level support into a global score. Explanations use the same evidence and decisions. We train Qwen3.5-based 4B and 9B models on 100K samples with multi-task SFT, counterfactual refinement, and evidence-aware GDPO. Across eight benchmarks, VisExMEM-9B achieves the highest descriptive composite among open-source evaluators in our comparison, exceeding VQAScore by 11.7 points while remaining competitive with GPT-5.6 Sol. We also introduce AGITA-Bench, containing 600 human-annotated image-text pairs and 8.7K atomic claims for global and claim-level evaluation. For untuned Qwen3.5 and GPT-5.6 Sol controls, decomposed prompting raises the same composite by 3.6-5.1 points relative to holistic prompting. Human evaluation shows that VisExMEM explanations outperform prior explainable metrics and approach GPT-5.6 Sol.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2610.16441 [cs.CV]
(or arXiv:2610.16441v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2610.16441 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 272 words
Related coverage
[AINews] not much happened today
Latent Space · Sep 10, 2026
Hands-free first notice of loss: Using Strands Agents and Amazon Bedrock AgentCore Browser Tool for intelligent claims intake
AWS · Jun 9, 2026
Why do models task game?
AI Alignment Forum · Aug 6, 2026
Debating RSI, the US-China Gap, and Jaggedness with JS Denain of Epoch AI
Interconnects · Sep 22, 2026