MESSY STREETS: A Benchmark for Geocoding Real-World Addresses

arXiv:2609.01612v1 Announce Type: cross Abstract: We introduce MESSY STREETS, a benchmark for evaluating geocoders on verbatim web addresses, with existence verification and controlled measurement of surface-form divergence. Unlike conventional benchmarks based on clean or synthetically perturbed addresses, MESSY STREETS contains addresses whose surface forms diverge from canonical representations and whose components may be missing, repeated, malformed, or incomplete. The benchmark is constructed from the December 2024 Web Data Commons corpus, with reference locations established from OpenAddresses or OpenStreetMap. The strongest commercial geocoders outperform open-source systems by up to 49 percentage points in recall. This gap is driven primarily by differences in candidate return rates on non-canonical addresses; once a candidate is returned, positional accuracy is broadly comparable across systems. Non-canonical surface form alone accounts for up to 25 percentage points of recall loss. Examining Nominatim's query-processing pipeline, we show that its conjunctive matching lets a single unrecognised token zero an otherwise valid query. The results demonstrate that geocoder choice is a consequential design decision for applications processing noisy address data, and that normalisation and preprocessing could substantially narrow the gap between open-source and commercial geocoders.

Abstract:We introduce MESSY STREETS, a benchmark for evaluating geocoders on verbatim web addresses, with existence verification and controlled measurement of surface-form divergence. Unlike conventional benchmarks based on clean or synthetically perturbed addresses, MESSY STREETS contains addresses whose surface forms diverge from canonical representations and whose components may be missing, repeated, malformed, or incomplete. The benchmark is constructed from the December 2024 Web Data Commons corpus, with reference locations established from OpenAddresses or OpenStreetMap. The strongest commercial geocoders outperform open-source systems by up to 49 percentage points in recall. This gap is driven primarily by differences in candidate return rates on non-canonical addresses; once a candidate is returned, positional accuracy is broadly comparable across systems. Non-canonical surface form alone accounts for up to 25 percentage points of recall loss. Examining Nominatim's query-processing pipeline, we show that its conjunctive matching lets a single unrecognised token zero an otherwise valid query. The results demonstrate that geocoder choice is a consequential design decision for applications processing noisy address data, and that normalisation and preprocessing could substantially narrow the gap between open-source and commercial geocoders.

Information Retrieval (cs.IR); Computation and Language (cs.CL)

arXiv:2609.01612 [cs.IR]

(or arXiv:2609.01612v1 [cs.IR] for this version)

https://doi.org/10.48550/arXiv.2609.01612 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

MESSY STREETS: A Benchmark for Geocoding Real-World Addresses

Related coverage