aiqbench analysis
This research establishes a rigorous mathematical proof that the cumulative prediction error during LLM autoregressive decoding forms a martingale, linking sequence fluctuations directly to predictive uncertainty. By deriving closed-form expressions for second-order behavior validated across models ranging from 1B to 72B parameters, the study provides a theoretical foundation for uncertainty quantification and text detection.
Autoregressive Decoding as a Martingale: Characterizing the Stochastic Dynamics of LLM Generation
arXiv:2610.16041v1 Announce Type: cross Abstract: Large language models (LLMs) have achieved remarkable success across a wide range of language tasks. At their core, autoregressive LLMs are conditional probability models that learn to approximate the next-token distribution given the preceding context. Despite this probabilistic nature, the stochastic dynamics of autoregressive decoding have rarely been investigated from a rigorous probabilistic perspective. In this work, we define the per-step prediction error as the difference between the sampled token indicator and the model's conditional predictive distribution, and prove that the cumulative error process is a martingale with respect to the decoding filtration, implying that its conditional first-order moment is identically zero. Beyond this first-order property, we derive a closed-form expression for the second-order behavior of the martingale sequence: its predictable quadratic variation reduces to $\sum_t\left(1-p(y_t\mid y_{<t})\right)$, computable from realized token probabilities alone. This identity establishes a direct theoretical connection between sequence fluctuations and predictive uncertainty during autoregressive decoding. White-box experiments on Llama-3.x and Qwen2.5 families (1B to 72B parameters) verify that the empirically measured second-order moments closely match the theoretical predictions derived from our martingale analysis. These results provide a probabilistic framework for analyzing autoregressive LLM decoding, reveal intrinsic probabilistic fingerprints of different language models, and establish a rigorous mathematical foundation for LLM-generated text detection and uncertainty quantification.
Abstract:Large language models (LLMs) have achieved remarkable success across a wide range of language tasks. At their core, autoregressive LLMs are conditional probability models that learn to approximate the next-token distribution given the preceding context. Despite this probabilistic nature, the stochastic dynamics of autoregressive decoding have rarely been investigated from a rigorous probabilistic perspective. In this work, we define the per-step prediction error as the difference between the sampled token indicator and the model's conditional predictive distribution, and prove that the cumulative error process is a martingale with respect to the decoding filtration, implying that its conditional first-order moment is identically zero. Beyond this first-order property, we derive a closed-form expression for the second-order behavior of the martingale sequence: its predictable quadratic variation reduces to $\sum_t\left(1-p(y_t\mid y_{<t})\right)$, computable from realized token probabilities alone. This identity establishes a direct theoretical connection between sequence fluctuations and predictive uncertainty during autoregressive decoding. White-box experiments on Llama-3.x and Qwen2.5 families (1B to 72B parameters) verify that the empirically measured second-order moments closely match the theoretical predictions derived from our martingale analysis. These results provide a probabilistic framework for analyzing autoregressive LLM decoding, reveal intrinsic probabilistic fingerprints of different language models, and establish a rigorous mathematical foundation for LLM-generated text detection and uncertainty quantification.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.16041 [cs.LG]
(or arXiv:2610.16041v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.16041 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 306 words
Related coverage
Stochastic Primal-Dual Decoding for Multiobjective Generative Recommender Systems
arXiv · Jul 23, 2026
CURE: Local Uncertainty Repair for Block-Parallel Speculative Decoding
arXiv · Aug 5, 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · Sep 29, 2026
AlignPoison: Conditional Targeted Poisoning of Generative Recommenders via Decoding-Aligned Prefixes
arXiv · Oct 15, 2026