aiqbench analysis
DEFAME introduces a modular, zero-shot multimodal large language model pipeline that dynamically selects tools and search depths to verify text-image claims without relying solely on parametric knowledge. By outperforming GPT-4o baselines on benchmarks with post-cutoff data, the system demonstrates strong temporal generalizability for real-time disinformation detection. This work advances the state-of-the-art in open-domain fact-checking by integrating visual evidence into structured, explainable reports.
DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts
arXiv:2412.10510v5 Announce Type: replace-cross Abstract: The proliferation of disinformation demands reliable and scalable fact-checking solutions. We present Dynamic Evidence-based FAct-checking with Multimodal Experts (DEFAME), a modular, zero-shot MLLM pipeline for open-domain, text-image claim verification. DEFAME operates in a six-stage process, dynamically selecting the tools and search depth to extract and evaluate textual and visual evidence. Unlike prior approaches that are text-only, lack explainability, or rely solely on parametric knowledge, DEFAME performs end-to-end verification, accounting for images in claims and evidence while generating structured, multimodal reports. Evaluation on the popular benchmarks VERITE, AVerITeC, and MOCHEG shows that DEFAME surpasses all previous methods, establishing itself as the new state-of-the-art fact-checking system for uni- and multimodal fact-checking. Moreover, we introduce a new multimodal benchmark, ClaimReview2024+, featuring claims after the knowledge cutoff of GPT-4o, avoiding data leakage. Here, DEFAME drastically outperforms the GPT-4o baselines, showing temporal generalizability and the potential for real-time fact-checking.
Abstract:The proliferation of disinformation demands reliable and scalable fact-checking solutions. We present Dynamic Evidence-based FAct-checking with Multimodal Experts (DEFAME), a modular, zero-shot MLLM pipeline for open-domain, text-image claim verification. DEFAME operates in a six-stage process, dynamically selecting the tools and search depth to extract and evaluate textual and visual evidence. Unlike prior approaches that are text-only, lack explainability, or rely solely on parametric knowledge, DEFAME performs end-to-end verification, accounting for images in claims and evidence while generating structured, multimodal reports. Evaluation on the popular benchmarks VERITE, AVerITeC, and MOCHEG shows that DEFAME surpasses all previous methods, establishing itself as the new state-of-the-art fact-checking system for uni- and multimodal fact-checking. Moreover, we introduce a new multimodal benchmark, ClaimReview2024+, featuring claims after the knowledge cutoff of GPT-4o, avoiding data leakage. Here, DEFAME drastically outperforms the GPT-4o baselines, showing temporal generalizability and the potential for real-time fact-checking.
ICML 2025 version. 9 pages main paper, 35 pages with appendix, 18 figures and 7 tables. v5: minor wording fixes and updated references. Code and benchmark: this https URL
Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
arXiv:2412.10510 [cs.CV]
(or arXiv:2412.10510v5 [cs.CV] for this version)
Full text under the publisher's open licence · 191 words
Related coverage
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026
MedPAIR: Measuring Whether Physicians and AI Agree on What Matters in Medical QA
Snorkel AI Blog · Oct 2, 2026
Hint-based CoT faithfulness evals still mostly work on Claude
LessWrong · Jul 30, 2026