analiza aiqbench
WildHandBench reveals a critical gap in current multimodal large language models, showing they struggle significantly with handwritten documents due to an over-reliance on linguistic priors rather than visual evidence. The new benchmark demonstrates that while humans still hold a narrow advantage, model failures are qualitatively distinct from human errors, highlighting the need for better evaluation metrics beyond simple accuracy scores. This research advances the field of computer vision and document intelligence by exposing systematic biases in how AI interprets unstructured handwritten data.
WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans
arXiv:2608.22959v1 Announce Type: cross Abstract: While the top model on OmniDocBench now reaches 96.34% overall on printed-document parsing, the ability of current models to handle challenging handwritten documents remains largely uncharacterized. Existing benchmarks focus on isolated text or formulas, overlook handwritten tables and real-world degradation, and report aggregate accuracy without explaining why models fail. We present WildHandBench, a benchmark containing 500 handwritten documents across three structures (free text, tables, formulas), four languages, and nine real-world scenarios. We introduce a Prior-Driven Error (PDE) metric that quantifies whether errors originate from language priors rather than visual evidence. Evaluating 18 state-of-the-art models together with calibrated human baselines, we find: (1) the best model achieves only 71.85% overall; (2) humans outperform all models yet the gap is narrow (77.09% vs. 71.85%); and (3) model errors are qualitatively different from human errors -- 63-91% of model errors are prior-driven versus only 49% for humans, exposing systematic reliance on language priors that conventional accuracy metrics cannot capture.
Abstract:While the top model on OmniDocBench now reaches 96.34% overall on printed-document parsing, the ability of current models to handle challenging handwritten documents remains largely uncharacterized. Existing benchmarks focus on isolated text or formulas, overlook handwritten tables and real-world degradation, and report aggregate accuracy without explaining why models fail. We present WildHandBench, a benchmark containing 500 handwritten documents across three structures (free text, tables, formulas), four languages, and nine real-world scenarios. We introduce a Prior-Driven Error (PDE) metric that quantifies whether errors originate from language priors rather than visual evidence. Evaluating 18 state-of-the-art models together with calibrated human baselines, we find: (1) the best model achieves only 71.85% overall; (2) humans outperform all models yet the gap is narrow (77.09% vs. 71.85%); and (3) model errors are qualitatively different from human errors -- 63-91% of model errors are prior-driven versus only 49% for humans, exposing systematic reliance on language priors that conventional accuracy metrics cannot capture.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2608.22959 [cs.CV]
(or arXiv:2608.22959v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2608.22959 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 256 cuvinte
Subiecte conexe
OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding
arXiv · 14 oct. 2026
OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios
arXiv · 21 aug. 2026
Do MLLMs Really Understand Low-Resource Khmer Documents? A Pilot Study on Khmer Document VQA
arXiv · 1 sept. 2026
Watch, Remember, Reason: Human-View Video Understanding with MLLMs
arXiv · 8 iun. 2026