aiqbench analysis
This research addresses the geometric instability caused by standard reinforcement learning updates in frozen embedding spaces, proposing a selective optimization method that preserves semantic manifolds while improving retrieval quality. By applying gradients only to positive advantages, the approach effectively aligns retrievers with downstream reranker preferences without disrupting pre-trained structures. The findings offer a practical solution for industrial-scale e-commerce systems seeking to enhance end-to-end retrieval performance through adaptive fine-tuning.
Learning from What You Retrieve: Online RL Fine-Tuning for Semantic Retrieval
arXiv:2608.30753v1 Announce Type: cross Abstract: In large-scale e-commerce retrieval, dual-encoder retrievers are op- timized for contrastive similarity, whereas downstream rerankers capture finer-grained relevance preferences; this objective mis- match limits end-to-end retrieval quality. Reinforcement Learning offers a way to use reward-model feedback for retriever adaptation, but we observe that standard policy-gradient updates can degrade embedding geometry, especially when the document index must remain frozen due to industrial constraints. To address this, we propose PAO (Positive-Advantage-Only), a selective RL optimization method. Our analysis reveals that in- discriminate penalization of negative samples (pushing away) in a frozen high-dimensional space disrupts pre-trained semantic man- ifolds. PAO selectively applies gradient updates only to retrieved items with positive advantages, effectively pulling query embed- dings toward high-reward regions while preserving global topo- logical stability. Experiments on both a massive industrial dataset and public benchmarks demonstrate that PAO significantly outper- forms standard RL and distillation baselines.
Abstract:In large-scale e-commerce retrieval, dual-encoder retrievers are op- timized for contrastive similarity, whereas downstream rerankers capture finer-grained relevance preferences; this objective mis- match limits end-to-end retrieval quality. Reinforcement Learning offers a way to use reward-model feedback for retriever adaptation, but we observe that standard policy-gradient updates can degrade embedding geometry, especially when the document index must remain frozen due to industrial constraints. To address this, we propose PAO (Positive-Advantage-Only), a selective RL optimization method. Our analysis reveals that in- discriminate penalization of negative samples (pushing away) in a frozen high-dimensional space disrupts pre-trained semantic man- ifolds. PAO selectively applies gradient updates only to retrieved items with positive advantages, effectively pulling query embed- dings toward high-reward regions while preserving global topo- logical stability. Experiments on both a massive industrial dataset and public benchmarks demonstrate that PAO significantly outper- forms standard RL and distillation baselines.
Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
arXiv:2608.30753 [cs.IR]
(or arXiv:2608.30753v1 [cs.IR] for this version)
https://doi.org/10.48550/arXiv.2608.30753 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 240 words
Related coverage
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · Jun 12, 2026
From transcription to semantic corpus analysis: unsupervised learning of sentence representations for ancient languages
arXiv · Jul 28, 2026
Presentation: Building Evals for AI Adoption: from Principles to Practice
InfoQ · May 29, 2026
Fast and Accurate: Improving Personalized LLM Performance by Semantic Concentration Inspired Workload Split between RAG and Fine-Tuning
arXiv · Oct 14, 2026