ARCH-B: Architectural Representation, Comprehension and Hierarchy Benchmark

Abstract:Multimodal models increasingly interpret visual environments, but their ability to recognize the same building across photographs, floor plans, elevations, sections, and renderings remains poorly characterized. We introduce ARCH-B, a benchmark of 354 four-choice questions across 11 cross-representational archetypes, constructed from a building-linked corpus of 3.9 million architectural images using visually similar distractors, model-guided difficulty screening, and manual validation. We evaluate 25 multimodal models and collect 5,830 responses from non-expert human participants. Model accuracy ranges from 10.45% to 83.90%, compared with a human baseline of 35.35%. Models perform comparatively well on mixed-representation outlier detection and photograph matching, but remain weaker on floorplan-to-photograph correspondence. Human and model difficulty across archetypes is only weakly correlated (Spearman's (\rho=0.33)). Held-out evaluation confirms that the difficulty identified during screening generalizes beyond the curation models. ARCH-B provides a diagnostic evaluation of visual correspondence and representation transfer across architectural media.

Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)

arXiv:2609.34047 [cs.CV]

(or arXiv:2609.34047v1 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2609.34047 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

benchmark.schema.json

evaluation_metadata.csv

runner/env.example

runner/llm/anthropic.go

runner/llm/bedrock.go

runner/llm/gemini.go

runner/llm/llm_client.go

runner/llm/openai.go

runner/load_questions.go

runner/questionRunner/benchmark.go

runner/questionRunner/csv_output.go

runner/questionRunner/image_utils.go

runner/questionRunner/types.go

runner/types/benchmark_result.go

runner/types/multi_benchmark_result.go

runner/types/question.go

(19 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.