aiqbench analysis
Ishigaki-IDS-Bench introduces a specialized evaluation framework for generating Building Information Modeling specifications, addressing the unique challenges of vocabulary conformance and external validator agreement that general benchmarks miss. In zero-shot testing across ten models, performance remains constrained with the best results achieving only 65.6% facet fidelity and a 33.1% content pass rate, highlighting significant gaps in current LLMs for this domain-specific structured generation task. The release of this dataset and evaluation code provides a necessary baseline for advancing AI applications in complex engineering workflows.
Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
arXiv:2605.22079v2 Announce Type: replace Abstract: Building Information Modeling (BIM) projects increasingly use Information Delivery Specification (IDS) to formalize information requirements in a machine-checkable XML format. Because IDS conditions are grounded in the Industry Foundation Classes (IFC) vocabulary, authoring them requires expertise in IFC concepts, validation tools, and property set conventions. Existing benchmarks for structured generation do not adequately capture the additional burden of vocabulary conformance and external-validator agreement that IDS imposes. We present Ishigaki-IDS-Bench, the first publicly released benchmark for IDS generation from BIM information requirements. The benchmark contains 166 examples spanning 83 practical scenarios authored in Japanese and English by six BIM/IDS experts, each paired with a gold IDS file and metadata covering input format, turn setting, target IFC versions, and construction domain. Evaluation proceeds in two stages: (i) formal validity scored by the buildingSMART IDSAuditTool along Processability, Structure, and Content, and (ii) content fidelity scored by facet-level macro-F1 against the gold IDS. Across 10 LLMs in zero-shot, the highest Facet F1 is 65.6%, achieved by GPT-5.5, while the highest Content pass rate is only 33.1%, achieved by Claude Opus 4.5. Ishigaki-IDS-Bench is released on Hugging Face (DOI 10.57967/hf/8873) under CC BY 4.0, and the evaluation code is released on Zenodo (DOI 10.5281/zenodo.20550510) under Apache-2.0.
Abstract:Building Information Modeling (BIM) projects increasingly use Information Delivery Specification (IDS) to formalize information requirements in a machine-checkable XML format. Because IDS conditions are grounded in the Industry Foundation Classes (IFC) vocabulary, authoring them requires expertise in IFC concepts, validation tools, and property set conventions. Existing benchmarks for structured generation do not adequately capture the additional burden of vocabulary conformance and external-validator agreement that IDS imposes. We present Ishigaki-IDS-Bench, the first publicly released benchmark for IDS generation from BIM information requirements. The benchmark contains 166 examples spanning 83 practical scenarios authored in Japanese and English by six BIM/IDS experts, each paired with a gold IDS file and metadata covering input format, turn setting, target IFC versions, and construction domain. Evaluation proceeds in two stages: (i) formal validity scored by the buildingSMART IDSAuditTool along Processability, Structure, and Content, and (ii) content fidelity scored by facet-level macro-F1 against the gold IDS. Across 10 LLMs in zero-shot, the highest Facet F1 is 65.6%, achieved by GPT-5.5, while the highest Content pass rate is only 33.1%, achieved by Claude Opus 4.5. Ishigaki-IDS-Bench is released on Hugging Face (DOI https://doi.org/10.57967/hf/8873) under CC BY 4.0, and the evaluation code is released on Zenodo (DOI https://doi.org/10.5281/zenodo.20550510) under Apache-2.0.
Computation and Language (cs.CL)
arXiv:2605.22079 [cs.CL]
(or arXiv:2605.22079v2 [cs.CL] for this version)
Full text under the publisher's open licence · 213 words
Related coverage
Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
arXiv · Aug 26, 2026
How ONESTRUCTION built the Ishigaki-IDS foundation model with AWS GenAIIC
AWS · Aug 11, 2026
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026
Ishigaki-IDS: An Open-Weight Verifier-Aware Model for Information Delivery Specification Drafting in Building Information Modeling
arXiv · Jun 9, 2026