Sycophants in the Courtroom: Are LLMs Fragile to Juridical Authority and Evolving Legal Standards?

Abstract:In medicine, claims remain valid when supported by empirical evidence grounded in stable biological reality. In law, by contrast, truth is contingent, defined by jurisdiction, temporal validity, and the hierarchy of authoritative sources. The recent success of large language models (LLMs) on medical licensing examinations has encouraged an expectation of comparable legal competence. This analogy, however, obscures a critical distinction between domains. Unlike in medicine, legal performance often depends less on inference than on determining when external authority is applicable, valid, and non-contradictory. We introduce a comparative diagnostic framework evaluating legal reasoning against medical baselines along four axes (knowledge recall, grounding, confidence, and robustness), uncovering a sharp domain asymmetry when applied to a new benchmark that encodes temporal validity and normative relationships. While medical LLMs reliably benefit from verified sources, legal LLMs struggle to assess when retrieved citations are useful or misleading, exhibiting overconfidence in perturbed contexts and sensitivity to superficial formatting cues. Increased model scale amplifies this tendency, revealing that stronger instruction following can coincide with weaker resistance to authoritative perturbations. These findings show that LLMs treat law as unstructured text rather than binding precedent, while revealing a tendency to over-trust authoritative but false information when external references conflict with a model's internal knowledge.

Please cite the definitive, peer-reviewed version of this article published in the Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), edited by Maria Liakata et al., Association for Computational Linguistics, pp. 10865-10886, 2026. DOI: this https URL

Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)

arXiv:2608.21409 [cs.CY]

(or arXiv:2608.21409v1 [cs.CY] for this version)

https://doi.org/10.48550/arXiv.2608.21409 arXiv-issued DOI via DataCite (pending registration)

Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, California, United States, Association for Computational Linguistics, 2026, pp. 10865-10886