analiza aiqbench
GeoOutageBench establishes a critical new standard for evaluating Large Language Models in interpreting ambiguous, multimodal geospatiotemporal data essential for power grid resilience. By integrating visual, textual, and structured outage records with domain ontologies, this benchmark directly addresses the limitations of existing Web-centric KGQA systems in handling real-world infrastructure scenarios. The release provides a necessary foundation for advancing LLM-KG architectures capable of robust spatiotemporal reasoning under complex environmental conditions.
GeoOutageBench: Benchmarking Ambiguity-aware, Ontology-grounded Geospatiotemporal KGQA for Multimodal Power Outage and Resilience Analysis
arXiv:2609.36082v1 Announce Type: cross Abstract: We introduce GeoOutageBench, a benchmark for assessing LLM-based geospatiotemporal KGQA for multimodal outage and resilience analysis. Unlike existing KGQA benchmarks for Web knowledge, GeoOutageBench considers a spatiotemporal KG that integrates visual, textual, and structured data from outage records, remote sensing, weather observations, storm and power events, geographic entities, and domain ontologies. It provides a competency query taxonomy at different difficulty levels from spatiotemporal containment and proximity, spatiotemporal co-occurrence analysis, multimodal evidence, to hypothetical evaluation. Over multimodal KG and query classes, GeoOutageBench provides user-configurable evaluation of three important, highly coherent yet less studied tasks: (1) LLMs' understanding for ambiguous geospatiotemporal questions in terms of NL to SPARQL interpretation, (2) query-driven assessment of ontology utility, and (3) answer accuracy of multimodal KGQA retrieval. GeoOutageBench provides a design principle and foundation for assessing LLM-KG systems that support real-world infrastructure resilience analysis. Our benchmark, source code, data, results, and other documentation are available at https://github.com/UCF-SAGE/GeoOutageBench.
Authors:Ethan D. Frakes, Amy Kvien, Rishabh Kundu, Redad Mehdi, Van D. Tran, Vibha S. Mandayam, Kristopher O. Davis, Erika I. Barcelos, Roger H. French, Yinghui Wu, Mengjie Li
Abstract:We introduce GeoOutageBench, a benchmark for assessing LLM-based geospatiotemporal KGQA for multimodal outage and resilience analysis. Unlike existing KGQA benchmarks for Web knowledge, GeoOutageBench considers a spatiotemporal KG that integrates visual, textual, and structured data from outage records, remote sensing, weather observations, storm and power events, geographic entities, and domain ontologies. It provides a competency query taxonomy at different difficulty levels from spatiotemporal containment and proximity, spatiotemporal co-occurrence analysis, multimodal evidence, to hypothetical evaluation. Over multimodal KG and query classes, GeoOutageBench provides user-configurable evaluation of three important, highly coherent yet less studied tasks: (1) LLMs' understanding for ambiguous geospatiotemporal questions in terms of NL to SPARQL interpretation, (2) query-driven assessment of ontology utility, and (3) answer accuracy of multimodal KGQA retrieval. GeoOutageBench provides a design principle and foundation for assessing LLM-KG systems that support real-world infrastructure resilience analysis. Our benchmark, source code, data, results, and other documentation are available at this https URL.
13 pages, 6 figures, 7 tables. Accepted to the 34th ACM International Conference on Advances in Geographic Information Systems (SIGSPATIAL '26), November 3-6, 2026, Riverside, CA, USA
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Information Retrieval (cs.IR)
arXiv:2609.36082 [cs.AI]
(or arXiv:2609.36082v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.36082 arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.1145/3841645.3842982 DOI(s) linking to related resources
Text integral sub licența deschisă a publicației · 239 cuvinte
Subiecte conexe
Neighborhood-Aware Dual Biomedical Entity Linking
arXiv · 6 aug. 2026
SatIR: Scalable High-Recall Constraint-Satisfaction-Based Information Retrieval for Clinical Trials Matching
arXiv · 9 iun. 2026
Genie One MCP: Give any AI Agent the Right Business Context
Databricks · 22 sept. 2026
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · 3 aug. 2026