Toward Measuring Structural Drift in LLM Communication Loops

arXiv:2604.13061v3 Announce Type: replace-cross Abstract: Large language models increasingly run in stateful pipelines that assemble each prompt from retrieval, memory, tools, and other agents. Such pipelines drift: information that should shape the next response is dropped, compressed, or misrouted while every component still reports success. Existing diagnostics miss this because they evaluate isolated prompts, responses, or task scores, whereas what decouples is the relation between a prompt and the response it draws. Here we show that treating the prompt to response to next prompt chain as the fundamental unit of analysis makes these relations measurable. We introduce structural communication coherence, quantified by two metrics: communication closure, which asks if what the pipeline returns at one turn matches what it faces next, and normalized conditional action contribution, which measures how much a sent message resolves the subsequent reply. Across 2,171 human to human, 58 human to LLM, and 8 LLM to LLM dialogues, these metrics reveal directional interaction structures; crucially, the measured contribution drops by 87 to 92% when a response is swapped for one from another turn, leaving surrounding prompts untouched. Because this approach requires no labels, healthy reference data, or predefined rules only the raw prompts and responses drift can be defined and measured directly from operational traffic, rather than inferred from eventual task failure. Establishing prospective detection performance is the next step.

Abstract:Large language models increasingly run in stateful pipelines that assemble each prompt from retrieval, memory, tools, and other agents. Such pipelines drift: information that should shape the next response is dropped, compressed, or misrouted while every component still reports success. Existing diagnostics miss this because they evaluate isolated prompts, responses, or task scores, whereas what decouples is the relation between a prompt and the response it draws. Here we show that treating the prompt to response to next prompt chain as the fundamental unit of analysis makes these relations measurable. We introduce structural communication coherence, quantified by two metrics: communication closure, which asks if what the pipeline returns at one turn matches what it faces next, and normalized conditional action contribution, which measures how much a sent message resolves the subsequent reply. Across 2,171 human to human, 58 human to LLM, and 8 LLM to LLM dialogues, these metrics reveal directional interaction structures; crucially, the measured contribution drops by 87 to 92% when a response is swapped for one from another turn, leaving surrounding prompts untouched. Because this approach requires no labels, healthy reference data, or predefined rules only the raw prompts and responses drift can be defined and measured directly from operational traffic, rather than inferred from eventual task failure. Establishing prospective detection performance is the next step.

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

arXiv:2604.13061 [cs.CL]

(or arXiv:2604.13061v3 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2604.13061 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Toward Measuring Structural Drift in LLM Communication Loops

Related coverage