aiqbench analysis
This article's primary subject is the theory of position bias in Transformer architectures, a machine learning research topic, so it passes the topic gate. Its core contribution is a structural explanation for Lost-in-the-Middle behavior based on causal masking, residual mixing, and positional encoding, with an adaptive rollout that tracks measured token influence across 30 model-dataset settings. Readers should weigh the reported ~85% reduction in profile discrepancy against the stated boundedness assumptions, since the theory's conclusions hold under those conditions and the empirical gains are relative to the authors' own baselines.
A Structural Theory of Position Bias in Transformers
arXiv:2602.16837v3 Announce Type: replace Abstract: Transformer models systematically favor certain token positions, yet the architectural origins of this position bias remain poorly understood. This bias is closely connected to the Lost-in-the-Middle phenomenon, where models underutilize information placed in the middle of the context. We show that causal Transformer architectures induce a systematic position-bias prior that can help explain the influence profiles associated with Lost-in-the-Middle. To characterize this prior, we develop a structural theory based on cumulative attention rollout with layerwise residual mixing. At finite depth, we isolate the competing positional drifts induced by causal masking, residual mixing, and positional encoding. Their interaction produces broad, often U-shaped profiles. At infinite depth, prior work predicts inevitable collapse under attention-only theory and identifies the discrepancy with observed Transformer behavior as an open problem. Under the stated boundedness assumptions, our residual-aware analysis replaces inevitable collapse with a dichotomy governed by cumulative attention mixing. Empirically, across 30 model-dataset settings, our adaptive residual-aware rollout closely matches measured input-token influence and reduces the average profile discrepancy by approximately 85% relative to both the attention-only and fixed-residual rollout baselines.
Abstract:Transformer models systematically favor certain token positions, yet the architectural origins of this position bias remain poorly understood. This bias is closely connected to the Lost-in-the-Middle phenomenon, where models underutilize information placed in the middle of the context. We show that causal Transformer architectures induce a systematic position-bias prior that can help explain the influence profiles associated with Lost-in-the-Middle. To characterize this prior, we develop a structural theory based on cumulative attention rollout with layerwise residual mixing. At finite depth, we isolate the competing positional drifts induced by causal masking, residual mixing, and positional encoding. Their interaction produces broad, often U-shaped profiles. At infinite depth, prior work predicts inevitable collapse under attention-only theory and identifies the discrepancy with observed Transformer behavior as an open problem. Under the stated boundedness assumptions, our residual-aware analysis replaces inevitable collapse with a dichotomy governed by cumulative attention mixing. Empirically, across 30 model-dataset settings, our adaptive residual-aware rollout closely matches measured input-token influence and reduces the average profile discrepancy by approximately 85% relative to both the attention-only and fixed-residual rollout baselines.
32 pages, 6 figures, 9 tables. Revised and expanded version with improved presentation, residual-aware finite- and infinite-depth theory, RoPE experiments, broader model and dataset coverage, and additional validation and ablation studies
Machine Learning (cs.LG)
arXiv:2602.16837 [cs.LG]
(or arXiv:2602.16837v3 [cs.LG] for this version)
Full text under the publisher's open licence · 218 words
Related coverage
Journey Operators for Structured Multi-Axis Composition
arXiv · Jul 31, 2026
Does preservation make sense before we know how to revive?
LessWrong · Jun 15, 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
Pattern Formation in Transformers
arXiv · Sep 30, 2026