analiza aiqbench
This arXiv paper benchmarks how well language models reproduce expert legal annotations on 471 U.S. civil-negligence opinions. Models do best on Facts and Outcome and weaker on Application of Law to Facts and Precedents, and the authors stress that coverage and cost matter alongside aggregate scores. For AI readers, the useful takeaway is that automated legal annotation looks feasible only with task-specific evaluation, and the reported results rest on the authors' own benchmark and cost estimates.
AnnoCaseLaw: Legal Annotation Benchmark of Civil Negligence
arXiv:2503.00128v2 Announce Type: replace-cross Abstract: Language models offer a way to scale costly, labour-intensive expert annotation, but their ability to reproduce expert judgements remains uncertain. This is particularly challenging in law, where annotators must distinguish the reasoning supporting judicial decisions from other legally relevant discussion. We present AnnoCaseLaw as a densely annotated benchmark comprising $471$ U.S. civil-negligence court opinions with more than $12,000$ expert highlights across five annotation categories. We evaluate proprietary frontier, open-weight, legal-specialised, and decision models on their ability to reproduce these annotations. Models perform better on Facts and Outcome than on Application of Law to Facts, Procedural History, and Relevant Precedents. On Application of Law to Facts, proprietary frontier models achieve the strongest balance of reasoning coverage and precision, while the open-weight model Gemma 4 achieves comparable Macro-F1 to proprietary frontier models at a fraction of the estimated inference cost. Our findings highlight the importance of evaluating annotation coverage and cost alongside aggregate scores when assessing the feasibility of automated legal annotation.
Abstract:Language models offer a way to scale costly, labour-intensive expert annotation, but their ability to reproduce expert judgements remains uncertain. This is particularly challenging in law, where annotators must distinguish the reasoning supporting judicial decisions from other legally relevant discussion. We present AnnoCaseLaw as a densely annotated benchmark comprising $471$ U.S. civil-negligence court opinions with more than $12,000$ expert highlights across five annotation categories. We evaluate proprietary frontier, open-weight, legal-specialised, and decision models on their ability to reproduce these annotations. Models perform better on Facts and Outcome than on Application of Law to Facts, Procedural History, and Relevant Precedents. On Application of Law to Facts, proprietary frontier models achieve the strongest balance of reasoning coverage and precision, while the open-weight model Gemma 4 achieves comparable Macro-F1 to proprietary frontier models at a fraction of the estimated inference cost. Our findings highlight the importance of evaluating annotation coverage and cost alongside aggregate scores when assessing the feasibility of automated legal annotation.
Revised version. AnnoCaseLaw is refocused from legal judgment prediction to expert legal annotation. The corpus and expert annotations are retained, while the annotation task, model evaluations, experiments, and analyses are revised and extended. The author list has been updated to reflect the contributions to this version
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2503.00128 [cs.CL]
(or arXiv:2503.00128v2 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 221 cuvinte