aiqbench analysis
This paper studies which signals predict whether hard-negative training data will help dense retrievers transfer to new tasks, before any training is done. Its main finding is that a simple IDF-weighted lexical measure, query-token noncoverage, reproduces the aggregate transfer ordering across backbones and settings, while several frozen-encoder semantic statistics do not. The authors also show that unrelated and false-negative passages can look favorable on these same statistics, so the signal describes source-level trends rather than the reliability of individual negatives.
What Tracks Hard-Negative Transfer? A Study of Lexical and Semantic Signals
arXiv:2603.20990v5 Announce Type: replace-cross Abstract: Which properties of hard-negative supervision track dense-retriever transfer before training? We compare lexical noncoverage, frozenencoder similarity statistics, hardness proxies, and combinations of these signals across BM25, dense, LLM-generated, and hybrid sources for MS MARCO training. Evaluation uses three retriever backbones and eleven BEIR datasets. Mean lexical residuality, an IDF-weighted measure of query-token noncoverage, reproduces the aggregate transfer ordering in all six backbone and setting blocks. The individual semantic statistics recover none of the complete orderings, and combining them with the lexical signal does not improve the observed rank agreement. After approximately matching BM25, dense, and LLM diagnostic samples for lexical residuality, none of the tested semantic statistics recovers the observed LLM > Dense > BM25 transfer ordering across three backbones, three matching resolutions, and 100 repetitions. Constructed controls further show that unrelated and false-negative-shaped passages can exhibit favorable lexical and semantic statistics. The findings identify a simple lexical association in this setting while distinguishing source level transfer agreement from the reliability of individual negatives.
Title:What Tracks Hard-Negative Transfer? A Study of Lexical and Semantic Signals
Abstract:Which properties of hard-negative supervision track dense-retriever transfer before training? We compare lexical noncoverage, frozenencoder similarity statistics, hardness proxies, and combinations of these signals across BM25, dense, LLM-generated, and hybrid sources for MS MARCO training. Evaluation uses three retriever backbones and eleven BEIR datasets. Mean lexical residuality, an IDF-weighted measure of query-token noncoverage, reproduces the aggregate transfer ordering in all six backbone and setting blocks. The individual semantic statistics recover none of the complete orderings, and combining them with the lexical signal does not improve the observed rank agreement. After approximately matching BM25, dense, and LLM diagnostic samples for lexical residuality, none of the tested semantic statistics recovers the observed LLM > Dense > BM25 transfer ordering across three backbones, three matching resolutions, and 100 repetitions. Constructed controls further show that unrelated and false-negative-shaped passages can exhibit favorable lexical and semantic statistics. The findings identify a simple lexical association in this setting while distinguishing source level transfer agreement from the reliability of individual negatives.
Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
arXiv:2603.20990 [cs.IR]
(or arXiv:2603.20990v5 [cs.IR] for this version)
https://doi.org/10.48550/arXiv.2603.20990 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 270 words
Related coverage
$\mathrm{ECI}_{\mathrm{sem}}$: Semantic Residual Effective Contrastive Information for Evaluating Hard Negatives
arXiv · Jun 8, 2026
ECI: Effective Contrastive Information to Evaluate Hard-Negatives
arXiv · Jun 6, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026