Skip to content
aiqbench takeAI Products

aiqbench analysis

This research demonstrates a critical limitation in current LLMs for legal applications: they often fail at the final step of reasoning by contradicting their own extracted facts when generating answers directly, whereas a hybrid approach using temporal dependency graphs and code achieves significantly higher accuracy. The study highlights that while models can extract dates correctly, relying on them to perform complex statutory calculations without structural constraints leads to verifiable errors in high-stakes deadline computations.
Sources coveringarXiv, LessWrong, Quantum Frontiers, AI Alignment Forum
What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

Time as Structure: Temporal Dependency Graphs for Verifiable Deadline Computation over Legal Documents

arXiv3 min read

arXiv:2608.15270v1 Announce Type: new Abstract: Miss a filing deadline by one day and the claim is barred, however strong the case. Computing that deadline is rarely simple: the period runs from a triggering event, is counted by a statutory convention, and may be suspended by a mandatory conciliation window. We ask whether a language model should answer such questions directly, or read the document and leave the arithmetic to code. We extract dated facts and their dependencies into a temporal dependency graph and compute deadlines from it with a calendar-correct engine. On UK Employment Appeal Tribunal judgments the engine reproduces six of seven timeliness rulings, and matches the judges' own dates to the day. The strongest of four language models, asked the same cases, gets the arithmetic right and the answer wrong: in six of twenty-one responses its stated verdict contradicts its own thinking, and every contradiction runs the same way, calling a late claim timely. To test the systems at scale we move the dismissal date across the statutory boundary, generating 427 cases whose answers are computed rather than annotated. On the cases both systems answer, the pipeline is right 90.2% of the time against 61.2% for direct answering. The limit is extraction: on contracts the errors are almost never in the arithmetic, but in choosing which event the period starts from.

Abstract:Miss a filing deadline by one day and the claim is barred, however strong the case. Computing that deadline is rarely simple: the period runs from a triggering event, is counted by a statutory convention, and may be suspended by a mandatory conciliation window. We ask whether a language model should answer such questions directly, or read the document and leave the arithmetic to code. We extract dated facts and their dependencies into a temporal dependency graph and compute deadlines from it with a calendar-correct engine. On UK Employment Appeal Tribunal judgments the engine reproduces six of seven timeliness rulings, and matches the judges' own dates to the day. The strongest of four language models, asked the same cases, gets the arithmetic right and the answer wrong: in six of twenty-one responses its stated verdict contradicts its own thinking, and every contradiction runs the same way, calling a late claim timely. To test the systems at scale we move the dismissal date across the statutory boundary, generating 427 cases whose answers are computed rather than annotated. On the cases both systems answer, the pipeline is right 90.2% of the time against 61.2% for direct answering. The limit is extraction: on contracts the errors are almost never in the arithmetic, but in choosing which event the period starts from.

13 pages, 2 figures, 5 tables. Preprint

Computation and Language (cs.CL)

I.2.7; I.2.4; J.1

arXiv:2608.15270 [cs.CL]

(or arXiv:2608.15270v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2608.15270 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Full text under the publisher's open licence · 323 words

Related coverage