analiza aiqbench
The paper introduces a benchmark of 83 published nonlinear works to test whether LLMs can tell stories out of chronological order. It finds that models default strongly toward chronological telling, and that readers find their nonlinear stories harder to follow, even when a structure is requested. Readers also preferred stories built from a causal story graph over those constrained to the published telling order, a difference the automated judge missed, so LLM-as-judge scores for narrative quality may be unreliable.
Can LLMs Tell Stories Out of Order? A Corpus and Benchmark for Nonlinear Narrative Generation
arXiv:2610.21468v1 Announce Type: cross Abstract: Every story has two orders: the order in which its events happen and the order in which they are told. A story is nonlinear when the two differ. We introduce a benchmark of 83 published works that measures whether language models can tell nonlinear stories. For each work, we annotate one list of events in both orders, together with a story graph of causal dependencies, character knowledge, and the questions the telling raises and resolves. Given events in chronological order, all 19 models tell them, on average, more chronologically than the published works do; asked for a specific structure, they land within the human range about half the time. We then compare a model's stories of the same events under different conditions, using both an automated judge and 71 human readers. Both prefer the chronological story a model writes by default to the nonlinear one it writes on request, which readers find harder to follow in 81% of decisive judgments. Readers also prefer stories written from the story graph to stories constrained to the published telling order (72% of decisive judgments), a difference the judge does not detect. Our results suggest that following an instruction to reorder a story is not yet the same as telling it well out of order.
Title:Can LLMs Tell Stories Out of Order? A Corpus and Benchmark for Nonlinear Narrative Generation
Abstract:Every story has two orders: the order in which its events happen and the order in which they are told. A story is nonlinear when the two differ. We introduce a benchmark of 83 published works that measures whether language models can tell nonlinear stories. For each work, we annotate one list of events in both orders, together with a story graph of causal dependencies, character knowledge, and the questions the telling raises and resolves. Given events in chronological order, all 19 models tell them, on average, more chronologically than the published works do; asked for a specific structure, they land within the human range about half the time. We then compare a model's stories of the same events under different conditions, using both an automated judge and 71 human readers. Both prefer the chronological story a model writes by default to the nonlinear one it writes on request, which readers find harder to follow in 81% of decisive judgments. Readers also prefer stories written from the story graph to stories constrained to the published telling order (72% of decisive judgments), a difference the judge does not detect. Our results suggest that following an instruction to reorder a story is not yet the same as telling it well out of order.
Under review, 36 pages, 13 figures, 15 tables. Code and data: this https URL. Project page: this https URL
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2610.21468 [cs.CL]
(or arXiv:2610.21468v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.21468 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 343 cuvinte
