aiqbench analysis
This paper presents "Policy Central," a deployed AI system designed to analyze police department policy documents for conflicts, redundancies, and similarities using two distinct architectural approaches. The study evaluates the trade-offs between a low-cost retrieval pipeline and a selective audit cascade, demonstrating that while current models effectively judge relationships once pairs are found, the primary challenge remains efficiently locating relevant document sections within large libraries. By significantly reducing the time required to scan thousands of policy sections, this system alters the economics of legal compliance review even before achieving perfect recall.
Policy Central: Detecting Conflicts, Redundancy, and Similarity in Deployed Police Policy
arXiv:2610.29338v1 Announce Type: new Abstract: Policy Central is a deployed system that helps police departments find relationships between sections of their written policy: whether two sections conflict, are redundant, are merely similar, or are different. We describe the system and compare two architectures. Retrieve-Judge, a retrieve-then-classify pipeline, runs as a cheap library-wide health signal; Screen-Confirm, a screen-then-confirm cascade, is a selective audit of the documents a reviewer chooses, checking them exhaustively against one another. On a 302-pair benchmark drawn from one department's ~84 real policy documents, given each pair Screen-Confirm reaches 0.795 aggregate recall and 0.885 on Conflict, so current models judge well. Finding the pair is the hard part: Retrieve-Judge's composed recall falls to 0.457, and Screen-Confirm recovers 0.472 (0.615 on Conflict). Today these pairs are found by reading the whole library, nearly 20,000 sections, checked against one another and re-checked after every revision. A scan that returns half of them in minutes, at low cost, and can be re-run on demand changes the economics of policy review even before recall improves.
Abstract:Policy Central is a deployed system that helps police departments find relationships between sections of their written policy: whether two sections conflict, are redundant, are merely similar, or are different. We describe the system and compare two architectures. Retrieve-Judge, a retrieve-then-classify pipeline, runs as a cheap library-wide health signal; Screen-Confirm, a screen-then-confirm cascade, is a selective audit of the documents a reviewer chooses, checking them exhaustively against one another. On a 302-pair benchmark drawn from one department's ~84 real policy documents, given each pair Screen-Confirm reaches 0.795 aggregate recall and 0.885 on Conflict, so current models judge well. Finding the pair is the hard part: Retrieve-Judge's composed recall falls to 0.457, and Screen-Confirm recovers 0.472 (0.615 on Conflict). Today these pairs are found by reading the whole library, nearly 20,000 sections, checked against one another and re-checked after every revision. A scan that returns half of them in minutes, at low cost, and can be re-run on demand changes the economics of policy review even before recall improves.
16 pages, 2 figures. Committed to EACL 2027
Computation and Language (cs.CL); Computers and Society (cs.CY)
arXiv:2610.29338 [cs.CL]
(or arXiv:2610.29338v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.29338 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 275 words
Related coverage
Effective Does Not Mean Useful: Conditional Functional Substitutability for Redundancy and Scaling in Transformers
arXiv · Oct 1, 2026
Beyond Effective Sample Size: Effective Number of Proposals for Adaptive Importance Sampling
arXiv · Aug 19, 2026
AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Models
arXiv · Jun 15, 2026
Beyond Similarity: Coverage-Aware Prompt Selection for Time Series Forecasting with LLMs
arXiv · Sep 22, 2026