aiqbench analysis
This study exposes critical label leakage in legal AI benchmarks derived from court registries, where simple keyword scans often outperform sophisticated models because metadata is inadvertently embedded in the input text. The research demonstrates that apparent model superiority frequently vanishes when label names are masked, revealing that many current evaluations measure data contamination rather than genuine reasoning capabilities. Consequently, the authors advocate for mandatory no-model baselines and masked controls to ensure fair and accurate assessment of legal AI systems.
Multi-Legal-Bench: When the Answer Is in the Input. Label Leakage in Legal Benchmarks Built from Court Registries
arXiv:2605.29738v3 Announce Type: replace Abstract: Court registries publish millions of decisions with structured metadata, which makes them an attractive source of labelled legal benchmarks: the court type, the form of the decision and its subject area come for free. We show that this convenience has a cost that registry-derived benchmarks rarely measure. We build Multi-Legal-Bench, which evaluates identical tasks on native court decisions from five national registries (France, the Netherlands, Poland, the Czech Republic and Lithuania), extending the Ukrainian UA-Legal-Bench, and audit what its cells actually measure. A keyword scan that uses no model reaches 96% on Dutch judgment-form classification against a 50% majority baseline, 84% on French court-type classification against 33%, and 66% on Polish judgment-form classification, where all nine models add at most seven points over it. Replacing the label names in the input by a mask brings the scan to the majority floor in all six affected cells, and eight of nine models lose accuracy significantly in most of them (39 of 54 model-cell pairs), by up to 48 points (Dutch judgment form: 100% to 52%); only Claude Sonnet 5 is essentially unaffected. Paired McNemar tests with Holm correction find a leader that beats every other model in only two of twelve cells, each a different model; but once the label names are masked, the same model leads both, and the spread between models widens in every masked cell. Part of the apparent parity between models is produced by the leakage itself. The audit also exposed a scoring defect in our own earlier release (answers written with diacritics were scored as wrong, understating Czech judgment-form accuracy by up to 90 points), which we correct and document. We recommend that every benchmark built from court registries report a no-model baseline and a masked control per cell. All data, prompts, predictions and scoring code are released.
Abstract:Court registries publish millions of decisions with structured metadata, which makes them an attractive source of labelled legal benchmarks: the court type, the form of the decision and its subject area come for free. We show that this convenience has a cost that registry-derived benchmarks rarely measure. We build Multi-Legal-Bench, which evaluates identical tasks on native court decisions from five national registries (France, the Netherlands, Poland, the Czech Republic and Lithuania), extending the Ukrainian UA-Legal-Bench, and audit what its cells actually measure. A keyword scan that uses no model reaches 96% on Dutch judgment-form classification against a 50% majority baseline, 84% on French court-type classification against 33%, and 66% on Polish judgment-form classification, where all nine models add at most seven points over it. Replacing the label names in the input by a mask brings the scan to the majority floor in all six affected cells, and eight of nine models lose accuracy significantly in most of them (39 of 54 model-cell pairs), by up to 48 points (Dutch judgment form: 100% to 52%); only Claude Sonnet 5 is essentially unaffected. Paired McNemar tests with Holm correction find a leader that beats every other model in only two of twelve cells, each a different model; but once the label names are masked, the same model leads both, and the spread between models widens in every masked cell. Part of the apparent parity between models is produced by the leakage itself. The audit also exposed a scoring defect in our own earlier release (answers written with diacritics were scored as wrong, understating Czech judgment-form accuracy by up to 90 points), which we correct and document. We recommend that every benchmark built from court registries report a no-model baseline and a masked control per cell. All data, prompts, predictions and scoring code are released.
28 pages, 4 figures, 21 tables. v3 rewrites the paper around label leakage in registry-derived benchmarks, adds a masked control for nine models, and corrects a scoring defect in v1/v2. Data, predictions and code: this https URL
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2605.29738 [cs.CL]
(or arXiv:2605.29738v3 [cs.CL] for this version)
Full text under the publisher's open licence · 354 words
Related coverage
[AINews] not much happened today
Latent Space · Oct 3, 2026
[AINews] Gemini 4 Argon: GDM’s answer to Astra/Fable, with 1M output
Latent Space · Oct 1, 2026
[AINews] not much happened today
Latent Space · Sep 18, 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026