aiqbench analysis
TechPulse take: This paper introduces $\mathrm{ECI}_{\mathrm{sem}}$, a training-free metric that leverages frozen target-encoder embeddings to rank hard-negative sources for dense retrieval without requiring downstream fine-tuning. By incorporating semantic locality and lexical residuality, the method effectively identifies high-value negative examples across various models, including DistilBERT and Contriever, while maintaining stability against perturbations in temperature and tokenizer choices. The approach offers a practical, theory-backed alternative for optimizing retrieval systems where post-hoc evaluation is currently the standard bottleneck.
$\mathrm{ECI}_{\mathrm{sem}}$: Semantic Residual Effective Contrastive Information for Evaluating Hard Negatives
arXiv:2603.20990v3 Announce Type: replace-cross Abstract: Hard-negative source selection for dense retrieval is usually decided only after fine-tuning and downstream evaluation. We propose $\mathrm{ECI}_{\mathrm{sem}}$, a semantic residual variant of Effective Contrastive Information (ECI) that ranks candidate negative sources using frozen target-encoder embeddings. $\mathrm{ECI}_{\mathrm{sem}}$ is training-free, not label-free: each scored example requires a query, a labeled positive, and an explicit candidate negative. $\mathrm{ECI}_{\mathrm{sem}}$ builds a weighted residual information matrix from target consistency, semantic locality, lexical residuality, and a log-determinant diversity objective. On MS MARCO negative sources, in-family $\mathrm{ECI}_{\mathrm{sem}}$ ranks LLM negatives highest among non-hybrid sources and Dense+LLM highest among hybrid sources, matching the strongest aggregate BEIR transfer results across DistilBERT, E5-base, and Contriever. Controlled ablations show that this alignment depends on using the target encoder family, while additional ablations show stability under sample-size, temperature, tokenizer, and IDF-corpus perturbations. The theory gives a local linearized link to loss reduction, while the empirical study treats downstream evaluation as the final test.
Abstract:Hard-negative source selection for dense retrieval is usually decided only after fine-tuning and downstream evaluation. We propose $\mathrm{ECI}_{\mathrm{sem}}$, a semantic residual variant of Effective Contrastive Information (ECI) that ranks candidate negative sources using frozen target-encoder embeddings. $\mathrm{ECI}_{\mathrm{sem}}$ is training-free, not label-free: each scored example requires a query, a labeled positive, and an explicit candidate negative. $\mathrm{ECI}_{\mathrm{sem}}$ builds a weighted residual information matrix from target consistency, semantic locality, lexical residuality, and a log-determinant diversity objective. On MS MARCO negative sources, in-family $\mathrm{ECI}_{\mathrm{sem}}$ ranks LLM negatives highest among non-hybrid sources and Dense+LLM highest among hybrid sources, matching the strongest aggregate BEIR transfer results across DistilBERT, E5-base, and Contriever. Controlled ablations show that this alignment depends on using the target encoder family, while additional ablations show stability under sample-size, temperature, tokenizer, and IDF-corpus perturbations. The theory gives a local linearized link to loss reduction, while the empirical study treats downstream evaluation as the final test.
Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
arXiv:2603.20990 [cs.IR]
(or arXiv:2603.20990v3 [cs.IR] for this version)
Full text under the publisher's open licence · 166 words
Related coverage
ECI: Effective Contrastive Information to Evaluate Hard-Negatives
arXiv · Jun 6, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026
Does Your LLM Trust You?
LessWrong · Aug 4, 2026
Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers
Hugging Face · Aug 26, 2026