aiqbench analysis
This research addresses the "Proximity Trap" in long-context LLMs, where models fail to retrieve distant evidence due to interference from irrelevant nearby text. The authors propose LYRA, a novel attention mechanism that reshapes context retrieval distributions to prioritize task-relevant information while preserving positional data. Their new benchmark, ProxBench, provides a rigorous framework for evaluating model performance under increasing levels of proximal background noise.
The Sirens' Song: When Proximal Background Context Overshadows Distant Evidence
arXiv:2609.26718v1 Announce Type: cross Abstract: Long-context LLMs focus on retrieving distant evidence from extensive context, yet existing work has largely focused on overcoming distance alone. In this work, we identify the Proximity Trap, insufficient attention to distant evidence often arises less from distance itself than from cumulative competition with abundant, task-irrelevant proximal background. To address the Proximity Trap, we introduce LYRA (Long-context heavY-tailed Relevance Alignment), a t-distributed directional matching mechanism that reshapes the context retrieval distribution, directing more attention mass toward task-relevant evidence, while preserving the relative positional information encoded. Extensive experiments on LongBench-v2, RULER, and LongBench demonstrate consistent improvements across context lengths and task categories. We further introduce ProxBench, a multi-level fine-grained benchmark for evaluating distant evidence utilization under increasing proximal background interference. Project page: https://xiaoyuyoung.github.io/LYRA/
Abstract:Long-context LLMs focus on retrieving distant evidence from extensive context, yet existing work has largely focused on overcoming distance alone. In this work, we identify the Proximity Trap, insufficient attention to distant evidence often arises less from distance itself than from cumulative competition with abundant, task-irrelevant proximal background. To address the Proximity Trap, we introduce LYRA (Long-context heavY-tailed Relevance Alignment), a t-distributed directional matching mechanism that reshapes the context retrieval distribution, directing more attention mass toward task-relevant evidence, while preserving the relative positional information encoded. Extensive experiments on LongBench-v2, RULER, and LongBench demonstrate consistent improvements across context lengths and task categories. We further introduce ProxBench, a multi-level fine-grained benchmark for evaluating distant evidence utilization under increasing proximal background interference. Project page: this https URL
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2609.26718 [cs.LG]
(or arXiv:2609.26718v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.26718 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 221 words
Related coverage
All the latest news on Meta’s cute, creepy Muse AI agent
The Verge · Sep 30, 2026
[AINews] OpenAI reports Navier-Stokes singularity find in 88 hours using Astra-next, roughly 10,000 agents and 130B tokens (>$40M), a contender for second ever Millennium Prize awarded
Latent Space · Sep 9, 2026
Do Large Language Models Play Six Degrees of Separation? Measuring Topological Compression in Long-Context Manifolds
arXiv · Aug 19, 2026
How do neural networks regress cubic polynomials? Apparently, they use a trick invented in Milan 500 years ago
LessWrong · Jul 22, 2026