analiza aiqbench
This research addresses a fundamental limitation in Rotary Positional Embedding (RoPE) regarding long-context modeling by introducing Token-Aware Phase Attention (TAPA). The proposed method integrates a learnable phase function directly into the attention mechanism to eliminate intrinsic distance-dependent biases without requiring post-hoc adjustments. Consequently, TAPA demonstrates superior performance in preserving token interactions over extended ranges and extrapolating to unseen sequence lengths compared to standard RoPE baselines.
Positional Encoding via Token-Aware Phase Attention
arXiv:2509.12635v4 Announce Type: replace Abstract: We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention scores that limits RoPE's ability to model long-context. RoPE extension methods may alleviate this issue, but they typically require post-hoc adjustments after pretraining, such as rescaling or hyperparameters retuning. This paper introduces Token-Aware Phase Attention (TAPA), a new positional encoding method that incorporates a learnable phase function into the attention mechanism. TAPA preserves token interactions over long range, extends to longer contexts with direct and light continual pretraining, extrapolates to unseen lengths, and attains substantially lower perplexity and stronger retrieval performance in the long-context regime than RoPE-style baselines.
Abstract:We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention scores that limits RoPE's ability to model long-context. RoPE extension methods may alleviate this issue, but they typically require post-hoc adjustments after pretraining, such as rescaling or hyperparameters retuning. This paper introduces Token-Aware Phase Attention (TAPA), a new positional encoding method that incorporates a learnable phase function into the attention mechanism. TAPA preserves token interactions over long range, extends to longer contexts with direct and light continual pretraining, extrapolates to unseen lengths, and attains substantially lower perplexity and stronger retrieval performance in the long-context regime than RoPE-style baselines.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2509.12635 [cs.CL]
(or arXiv:2509.12635v4 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2509.12635 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 201 cuvinte
Subiecte conexe
Remember to Forget: Gated Adaptive Positional Encoding
arXiv · 9 oct. 2026
MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation
arXiv · 2 sept. 2026
RIG-RoPE: Relation- and Instance-Gated Rotary Positional Encoding with Duration-Aware Temporal Coordinates
arXiv · 7 aug. 2026
Adaptive Parallel Reasoning: The Next Paradigm in Efficient Inference Scaling
BAIR Blog · 8 mai 2026