Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention

Abstract:Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not. This is a robust empirical regularity, and the explanations offered for it so far are chiefly about expressivity, that is, about whether a length-generalizing solution exists. We give an optimization explanation. On a minimal fixed-offset retrieval task that isolates positional selection, the gap is governed by the implicit bias of the trained attention head: among the many solutions that fit short sequences, which one gradient descent actually selects. We prove that rotary encodings make the attention logit a function of relative offset alone, an exact equivariance, so whatever selection rule is learned at training lengths is reproduced verbatim at every longer length. Learned absolute encodings instead leave out-of-range positions unconstrained, and the trained head pins to a fixed absolute position inside the training range. We characterize the learned rotary rule as a low-rank ``carrier'' kernel aligned with the target offset, and we derive the resulting graceful accuracy decay as an attention-dilution law; both predictions are confirmed across seeds and offsets. A linear-attention control shows the mechanism is specific to softmax: without normalization, training selects a min-norm interpolant that does not extrapolate. The phenomenon, the equivariance, and the carrier all transfer to a multi-layer, multi-head transformer trained on a full-sequence length-generalization task. The account connects the implicit bias of attention, implicit bias for extrapolation in recurrent models, and the learning side of the RASP-L conjecture.

Machine Learning (cs.LG)

arXiv:2607.18759 [cs.LG]

(or arXiv:2607.18759v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2607.18759 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

code_supplement/README.md

code_supplement/reproduce_all.sh

code_supplement/requirements.txt

code_supplement/results/RESULTS.md

code_supplement/results/base.log

code_supplement/results/frequency.log

code_supplement/results/frequency_rank.png

code_supplement/results/linear.log

code_supplement/results/linear_control.png

code_supplement/results/mechanics.log

code_supplement/results/mechanics.png

code_supplement/results/multiseed.log

code_supplement/results/multiseed_results.png

code_supplement/results/pe_variants.png

code_supplement/results/prop1_K1.log

code_supplement/results/prop1_K2.log

code_supplement/results/prop1_K3.log

code_supplement/results/prop1_validation_K1.png

code_supplement/results/prop1_validation_K2.png

code_supplement/results/prop1_validation_K3.png

code_supplement/results/shift_K1.log

code_supplement/results/shift_K2.log

code_supplement/results/shift_equivariance_K1.png

code_supplement/results/shift_equivariance_K2.png

code_supplement/results/transformer.log

code_supplement/results/transformer_shift.png

code_supplement/results/variants.log

code_supplement/rope_lengthgen.py

code_supplement/tests/test_smoke.py

(25 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.