aiqbench analysis
This research addresses a critical gap in AI by formalizing Legal Rule Induction, enabling Large Language Models to distill implicit doctrinal principles from judicial precedents rather than just applying existing statutes. By introducing the first benchmark for Chinese law comprising over 38,000 cases, the study demonstrates that current SOTA models suffer from over-generalization and hallucination, which are significantly reduced through targeted training on this new dataset. The work advances machine learning capabilities in extracting nuanced normative behaviors and legal consequences from unstructured case law across different jurisdictions.
Legal Rule Induction: Towards Generalizable Principle Discovery from Analogous Judicial Precedents
arXiv:2505.14104v2 Announce Type: replace Abstract: Legal rules encompass not only codified statutes but also implicit adjudicatory principles derived from precedents that contain discretionary norms, social morality, and policy. While computational legal research has advanced in applying established rules to cases, inducing legal rules from judicial decisions remains understudied across jurisdictions. The advent of Large Language Models (LLMs) offers unprecedented potential to automate the extraction of such latent principles, yet progress is stymied by the absence of formal task definitions and benchmarks. To address this gap, we formalize Legal Rule Induction (LRI) as the task of deriving concise, generalizable doctrinal rules from analogous precedents, distilling their shared preconditions, normative behaviors, and legal consequences. We further propose a reproducible pipeline for LRI dataset construction and, instantiating it on Chinese law as a representative jurisdiction, introduce the first LRI benchmark comprising 5,121 case sets (38,088 court cases) for model tuning and 216 expert-annotated gold test sets. Experimental results reveal that: 1) State-of-the-art LLMs struggle with over-generalization and hallucination; 2) Training on our dataset markedly enhances LLMs capabilities in capturing nuanced rule patterns across similar cases.
Abstract:Legal rules encompass not only codified statutes but also implicit adjudicatory principles derived from precedents that contain discretionary norms, social morality, and policy. While computational legal research has advanced in applying established rules to cases, inducing legal rules from judicial decisions remains understudied across jurisdictions. The advent of Large Language Models (LLMs) offers unprecedented potential to automate the extraction of such latent principles, yet progress is stymied by the absence of formal task definitions and benchmarks. To address this gap, we formalize Legal Rule Induction (LRI) as the task of deriving concise, generalizable doctrinal rules from analogous precedents, distilling their shared preconditions, normative behaviors, and legal consequences. We further propose a reproducible pipeline for LRI dataset construction and, instantiating it on Chinese law as a representative jurisdiction, introduce the first LRI benchmark comprising 5,121 case sets (38,088 court cases) for model tuning and 216 expert-annotated gold test sets. Experimental results reveal that: 1) State-of-the-art LLMs struggle with over-generalization and hallucination; 2) Training on our dataset markedly enhances LLMs capabilities in capturing nuanced rule patterns across similar cases.
Computation and Language (cs.CL)
arXiv:2505.14104 [cs.CL]
(or arXiv:2505.14104v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2505.14104 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 270 words
Related coverage
Deconfusing Corrigibility
AI Alignment Forum · Oct 17, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026
LiveLaw Supreme Court Half-Yearly Digest: January - June 2026 [Part 2]
livelaw.in · Sep 13, 2026