aiqbench analysis
The paper introduces a benchmark and baseline for testing whether vision-language models can perform measurement-style reasoning on satellite imagery, such as counting buildings or computing areas and distances. The best zero-shot end-to-end model scores 28.1%, while a program-writing agent (QASR) reaches 42.0%, and ablations show segmentation quality is its main bottleneck. For readers, the useful takeaway is that current VLMs remain weak at quantitative spatial tasks, and the released reference masks and training set make modular and end-to-end approaches easier to compare.
SQuID: A Benchmark for Quantitative Spatial Reasoning on Satellite Imagery
arXiv:2601.13401v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) describe satellite images fluently but struggle with the quantitative spatial reasoning that remote sensing (RS) analysts perform routinely, such as counting buildings, measuring forest area in hectares and computing distances in meters. We introduce SQuID (Satellite Quantitative Intelligence Dataset), a benchmark of 2,000 questions over 587 images from four sources at 0.3 to 1.0 m ground sampling distance, covering 24 question types in three difficulty tiers. Answers are computed from curated segmentation masks or, for 50 questions, from human consensus, and numeric answers are scored against tolerance ranges informed by human annotations. SQuID also releases the reference masks behind its answers, so that modular systems can be evaluated using reference rather than predicted masks, and 75,687 training question-answer pairs on disjoint images (SQuID-Train). Among 15 evaluated systems, the best zero-shot end-to-end VLM reaches 28.1%, and RS-specialized VLMs score lower. As a baseline, we propose QASR (Quantitative Agentic Spatial Reasoning), in which an LLM writes a program that segments the image and measures the masks with geometric operators. QASR reaches 42.0%. On the 1,950 mask-derived questions, running its programs on the reference masks instead of predicted masks raises accuracy from 42.2% to 82.6%, so segmentation is its main source of error. Fine-tuning Qwen2.5-VL-7B on SQuID-Train raises its accuracy from 16.1% to 40.7%, close to QASR. Multi-condition questions (Tier 3) remain the hardest in every setting, at 62.0% for QASR with reference masks and 13.2% after fine-tuning. These results leave clear room for end-to-end models that learn metric spatial reasoning, and SQuID provides the data to train and evaluate them.
Abstract:Vision-Language Models (VLMs) describe satellite images fluently but struggle with the quantitative spatial reasoning that remote sensing (RS) analysts perform routinely, such as counting buildings, measuring forest area in hectares and computing distances in meters. We introduce SQuID (Satellite Quantitative Intelligence Dataset), a benchmark of 2,000 questions over 587 images from four sources at 0.3 to 1.0 m ground sampling distance, covering 24 question types in three difficulty tiers. Answers are computed from curated segmentation masks or, for 50 questions, from human consensus, and numeric answers are scored against tolerance ranges informed by human annotations. SQuID also releases the reference masks behind its answers, so that modular systems can be evaluated using reference rather than predicted masks, and 75,687 training question-answer pairs on disjoint images (SQuID-Train). Among 15 evaluated systems, the best zero-shot end-to-end VLM reaches 28.1%, and RS-specialized VLMs score lower. As a baseline, we propose QASR (Quantitative Agentic Spatial Reasoning), in which an LLM writes a program that segments the image and measures the masks with geometric operators. QASR reaches 42.0%. On the 1,950 mask-derived questions, running its programs on the reference masks instead of predicted masks raises accuracy from 42.2% to 82.6%, so segmentation is its main source of error. Fine-tuning Qwen2.5-VL-7B on SQuID-Train raises its accuracy from 16.1% to 40.7%, close to QASR. Multi-condition questions (Tier 3) remain the hardest in every setting, at 62.0% for QASR with reference masks and 13.2% after fine-tuning. These results leave clear room for end-to-end models that learn metric spatial reasoning, and SQuID provides the data to train and evaluate them.
Revised version with updated title, author list, manuscript, and experiments. Submitted to NeurIPS 2026 Evaluations & Datasets Track. Introduces the SQuID benchmark and QASR baseline for quantitative spatial reasoning on satellite imagery. Dataset DOI: https://doi.org/10.57967/hf/7565. Code and data are publicly available
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2601.13401 [cs.CV]
(or arXiv:2601.13401v2 [cs.CV] for this version)
From: Peter A. Massih [view email] [v1] Mon, 19 Jan 2026 21:14:34 UTC (11,425 KB) [v2] Tue, 27 Oct 2026 14:45:28 UTC (13,248 KB)
Full text under the publisher's open licence · 343 words
Related coverage
USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environments
arXiv · Sep 30, 2026
MedVision: Benchmarking Quantitative Medical Image Analysis
arXiv · Jun 9, 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · Sep 29, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026