aiqbench analysis
This arXiv preprint asks whether supervised neural retrievers learn a query-independent "relevance prior" from biased annotation, and reports that lower-prior documents are harder to find even when relevant. The effect is stronger in dense retrievers than in BM25, which makes it relevant to anyone building or evaluating retrieval-augmented systems. Its central claim rests on classifier probes over frozen embeddings and LLM-generated explanations, so the findings are suggestive of a training-data bias rather than a settled mechanism.
Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors
arXiv:2606.02814v2 Announce Type: replace-cross Abstract: Neural retrievers are trained to estimate query-document relevance from annotated query-document pairs. Yet annotation protocols may not purely reflect relevance: they select only a subset of documents for labeling, and this selection can favor certain document types over others. We investigate whether supervised bi-encoder retrievers implicitly learn a document-level relevance prior: a query-independent signal encoded in their representation space as a side effect of training on annotated data. We estimate this prior by training simple classifiers on frozen document embeddings and evaluate three state-of-the-art retrievers across multiple IR benchmarks. We find that supervised neural retrievers encode relevance priors that generalize to unseen documents and are consistent across models. These priors are associated with a findability gap: documents with lower prior are systematically harder to retrieve, even when genuinely relevant. This effect appears in supervised dense retrievers but is weaker and less consistent in BM25, and it persists under controlled matched-document comparisons. Using LLM-based explanations, we find that judged-relevant documents tend to be comprehensive, self-contained summaries of mainstream topics, while niche, fragmentary, or highly technical content is often left unjudged. Retrievers appear to internalize this bias, ranking documents with these favored features higher than documents that lack them, independently of their actual relevance. Our findings expose a structural limitation of supervised retrieval: models trained on annotated data do not just learn relevance, but also the implicit document preferences in their training data.
Title:Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors
Abstract:Neural retrievers are trained to estimate query-document relevance from annotated query-document pairs. Yet annotation protocols may not purely reflect relevance: they select only a subset of documents for labeling, and this selection can favor certain document types over others. We investigate whether supervised bi-encoder retrievers implicitly learn a document-level relevance prior: a query-independent signal encoded in their representation space as a side effect of training on annotated data. We estimate this prior by training simple classifiers on frozen document embeddings and evaluate three state-of-the-art retrievers across multiple IR benchmarks. We find that supervised neural retrievers encode relevance priors that generalize to unseen documents and are consistent across models. These priors are associated with a findability gap: documents with lower prior are systematically harder to retrieve, even when genuinely relevant. This effect appears in supervised dense retrievers but is weaker and less consistent in BM25, and it persists under controlled matched-document comparisons. Using LLM-based explanations, we find that judged-relevant documents tend to be comprehensive, self-contained summaries of mainstream topics, while niche, fragmentary, or highly technical content is often left unjudged. Retrievers appear to internalize this bias, ranking documents with these favored features higher than documents that lack them, independently of their actual relevance. Our findings expose a structural limitation of supervised retrieval: models trained on annotated data do not just learn relevance, but also the implicit document preferences in their training data.
Information Retrieval (cs.IR); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2606.02814 [cs.IR]
(or arXiv:2606.02814v2 [cs.IR] for this version)
https://doi.org/10.48550/arXiv.2606.02814 Focus to learn more arXiv-issued DOI via DataCite
https://doi.org/10.18653/2026.findings-emnlp.2442 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 351 words
Related coverage
Introducing Embed 5—A new family of frontier embedding models
Cohere · Sep 30, 2026
VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval
arXiv · Oct 15, 2026
Document Optimization for Black-Box Retrieval via Reinforcement Learning
arXiv · Aug 6, 2026
Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors
arXiv · Jun 3, 2026