aiqbench analysis
Five LLMs were asked to reconstruct full Allen interval-relation networks over 360 narratives. Under the pre-registered parser, none beat an always-"before" baseline, but a post hoc parser recovering determined cells from malformed rows closes most of that gap, so output formatting drives much of the measured accuracy. Models handle precedence and containment but rarely overlap, and they struggle most with flashbacks. Logical consistency improves accuracy mainly by filling empty cells, so readers should treat it as a weak proxy for correctness.
Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives
arXiv:2610.17402v1 Announce Type: new Abstract: Understanding a narrative means knowing how every event relates in time to every other, yet large language models (LLMs) are mostly tested on isolated event pairs or coarse labels. We ask five LLMs to reconstruct the complete network of Allen's 13 interval relations for 360 English narratives whose events are placed relative to one another rather than by dates, and evaluate the outputs under a pre-registered protocol for agreement with our gold annotations, converse consistency and logical satisfiability. End to end, the best model clearly beats trivial baselines on macro-F1, our primary metric, yet under our pre-registered parser no model matches the accuracy of always answering "before", which the skewed labels inflate. Most of that accuracy gap is a parsing artifact: formatting errors in large matrices cost whole rows, and a post hoc parser that recovers the determined cells of malformed rows raises coverage from 0.60 to 0.84 and closes the gap to the baseline for the best model. On the cells they do answer, models recover precedence and containment but rarely overlap, and they fail most on flashbacks, where both the narration and the event list present the later event first. Logical consistency varies widely yet says little about agreement with gold: enforcing it raises accuracy mainly by filling empty cells, and is beaten there by the trivial control of guessing the majority label. Reliable structured output and robust handling of relative time are prerequisites for temporal reasoning over whole narratives.
Authors:Adib Sakhawat, Fardeen Sadab, Tamjid Hasan Fahim, Multazam Mahmud, Tahsan Ferdous Lihan, Md Isfak Iqbal Chowdhury, Nafis Ahnaf Jamil, Md. Atik Shahriar, Sieam Shahriare, Wasi Omar Syed, M Safwan Hasan Khan, Tabib Hassan
Abstract:Understanding a narrative means knowing how every event relates in time to every other, yet large language models (LLMs) are mostly tested on isolated event pairs or coarse labels. We ask five LLMs to reconstruct the complete network of Allen's 13 interval relations for 360 English narratives whose events are placed relative to one another rather than by dates, and evaluate the outputs under a pre-registered protocol for agreement with our gold annotations, converse consistency and logical satisfiability. End to end, the best model clearly beats trivial baselines on macro-F1, our primary metric, yet under our pre-registered parser no model matches the accuracy of always answering "before", which the skewed labels inflate. Most of that accuracy gap is a parsing artifact: formatting errors in large matrices cost whole rows, and a post hoc parser that recovers the determined cells of malformed rows raises coverage from 0.60 to 0.84 and closes the gap to the baseline for the best model. On the cells they do answer, models recover precedence and containment but rarely overlap, and they fail most on flashbacks, where both the narration and the event list present the later event first. Logical consistency varies widely yet says little about agreement with gold: enforcing it raises accuracy mainly by filling empty cells, and is beaten there by the trivial control of guessing the majority label. Reliable structured output and robust handling of relative time are prerequisites for temporal reasoning over whole narratives.
21 pages, 11 figures, 11 tables. Corpus, model outputs and analysis code available at this https URL
Computation and Language (cs.CL)
arXiv:2610.17402 [cs.CL]
(or arXiv:2610.17402v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.17402 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 311 words
Related coverage
WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms
arXiv · Jul 25, 2026
Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
arXiv · Oct 17, 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026
TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs
arXiv · Sep 9, 2026