analiza aiqbench
The paper argues that long-context retrieval failures in small language models (0.5B to 3B parameters) stem less from input length than from competition for attention. Length alone costs almost nothing, but distractor statements that offer rival values for the queried key degrade accuracy sharply, and attention mass on the evidence tracks the loss better than length does. The findings rest on small open models and a specific retrieval setup, so generalization to larger models and other tasks needs testing. The released probe and 126K trial-level measurements make the claims checkable, which matters for anyone evaluating long-context reliability or prior "length hurts" results.
Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval
arXiv:2611.04169v1 Announce Type: new Abstract: Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parameters, four families), neutral filler costs nothing: accuracy stays at 0.97 to 1.00, up to 32K tokens on Qwen2.5-1.5B. Statements in the format of the evidence, placed at the same positions, drop it to 0.16 to 0.33. A single statement that gives the queried key a rival value multiplies the loss by 9 to 110, and every additional rival adds to it. Inside the model, one quantity tracks the failure: the attention mass left on the evidence. Across 126,473 trials it explains up to 58% of the variance in loss; log length explains at most 8%. The masked-filler protocol behind the claim that length alone hurts reproduces on MMLU (11 to 18 points lost at 30K tokens), but fails a visible control in retrieval: masking costs 45 accuracy points at 32K tokens, while the same filler left visible costs none. In retrieval, context rot is a competition for attention. We release the probe and all 126K trial-level measurements.
Abstract:Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parameters, four families), neutral filler costs nothing: accuracy stays at 0.97 to 1.00, up to 32K tokens on Qwen2.5-1.5B. Statements in the format of the evidence, placed at the same positions, drop it to 0.16 to 0.33. A single statement that gives the queried key a rival value multiplies the loss by 9 to 110, and every additional rival adds to it. Inside the model, one quantity tracks the failure: the attention mass left on the evidence. Across 126,473 trials it explains up to 58% of the variance in loss; log length explains at most 8%. The masked-filler protocol behind the claim that length alone hurts reproduces on MMLU (11 to 18 points lost at 30K tokens), but fails a visible control in retrieval: masking costs 45 accuracy points at 32K tokens, while the same filler left visible costs none. In retrieval, context rot is a competition for attention. We release the probe and all 126K trial-level measurements.
15 pages, 2 figures, 5 tables. Code and trial-level data: this https URL
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2611.04169 [cs.CL]
(or arXiv:2611.04169v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2611.04169 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 301 cuvinte
Subiecte conexe
Attention as Conditioning: What Classical Learning Theory Predicts About Linear Transformers
arXiv · 1 sept. 2026
Retrieval Capacity of Self-Attention Under Competition
arXiv · 1 oct. 2026
TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention
arXiv · 8 iun. 2026
PANDA: Prior-guided Attentional Dual-path Architecture
arXiv · 12 oct. 2026