aiqbench analysis
This research identifies a critical bottleneck in Multimodal Large Language Models where geometric reasoning failures stem from poor visual perception rather than logical deficits. By introducing CapGeo-Bench with fine-grained keypoint metrics, the study provides a rigorous framework to evaluate and improve how MLLMs interpret diagrams and spatial relationships. This benchmark offers researchers a standardized tool to isolate and advance specific capabilities in multimodal geometry understanding.
CapGeo-Bench: Decoupling Visual Perception from Reasoning and Evaluating Geometric Understanding
arXiv:2510.09302v2 Announce Type: replace-cross Abstract: While Multimodal Large Language Models (MLLMs) have achieved remarkable success in difficult purely textual mathematical reasoning tasks, even advanced closed-source models such as GPT-o3 still struggle with geometric problems. This discrepancy motivates us to investigate the root cause: is the bottleneck of multimodal geometric reasoning rooted in reasoning itself, or in the perception of geometric information from diagrams? To answer this question, we conduct an exploratory experiment and find that providing high-quality captions consistently and substantially boosts performance of MLLMs, empirically validating the visual perception bottleneck in geometric reasoning. However, MLLMs' capabilities in visual geometric perception remain insufficiently evaluated. To fill the gap, we propose CapGeo-Bench, a benchmark of 4,641 high-quality figure-caption pairs equipped with a fine-grained keypoint-based evaluation metric. Specifically, CapGeo-Bench encompasses three geometric classes and is divided into four difficulty levels according to the complexity of geometric primitives. Captions are scored at a fine-grained level by extracting keypoints along three dimensions-geometric elements, spatial relationships, and numerical attributes. CapGeo-Bench provides a reliable instrument and guideline for evaluating and advancing MLLMs' geometric perception capabilities. Our code and data are publicly available at: https://github.com/YuYingLi0/CapGeo.
Abstract:While Multimodal Large Language Models (MLLMs) have achieved remarkable success in difficult purely textual mathematical reasoning tasks, even advanced closed-source models such as GPT-o3 still struggle with geometric problems. This discrepancy motivates us to investigate the root cause: is the bottleneck of multimodal geometric reasoning rooted in reasoning itself, or in the perception of geometric information from diagrams? To answer this question, we conduct an exploratory experiment and find that providing high-quality captions consistently and substantially boosts performance of MLLMs, empirically validating the visual perception bottleneck in geometric reasoning. However, MLLMs' capabilities in visual geometric perception remain insufficiently evaluated. To fill the gap, we propose CapGeo-Bench, a benchmark of 4,641 high-quality figure-caption pairs equipped with a fine-grained keypoint-based evaluation metric. Specifically, CapGeo-Bench encompasses three geometric classes and is divided into four difficulty levels according to the complexity of geometric primitives. Captions are scored at a fine-grained level by extracting keypoints along three dimensions-geometric elements, spatial relationships, and numerical attributes. CapGeo-Bench provides a reliable instrument and guideline for evaluating and advancing MLLMs' geometric perception capabilities. Our code and data are publicly available at: this https URL.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2510.09302 [cs.CV]
(or arXiv:2510.09302v2 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2510.09302 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 287 words
Related coverage
Why do models task game?
AI Alignment Forum · Aug 6, 2026
SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding
arXiv · Oct 10, 2026
Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?
arXiv · Jun 9, 2026
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026