aiqbench analysis
OntologyBench presents a critical evaluation of dense retrieval systems in biomedical contexts, revealing that current embedding and reranking methods struggle with complex relational and compositional tasks despite strong performance on simple concept grounding. The study demonstrates that while fine-tuning with ontology-derived supervision offers some gains, standard LLM-based scoring fails to end-to-end recover the structured compatibility encoded by medical ontologies. This work underscores the necessity for retrieval architectures that more deeply integrate learned representations with explicit biomedical knowledge graphs rather than relying solely on semantic similarity.
OntologyBench: Can Dense Retrieval Satisfy Structured Biomedical Constraints?
arXiv:2609.08174v1 Announce Type: new Abstract: We introduce OntologyBench, a tiered biomedical retrieval benchmark comprising 471,854 training and 125,744 evaluation query-document relevance pairs across concept grounding, relational retrieval, and compositional phenotype-based retrieval. Although these tasks can be tractable using ontology-aware reference methods, across task tiers, embedding performance is generally lower on relational and compositional tasks than on concept-grounding tasks. Fine-tuning on ontology-derived supervision improves performance on several relational and compositional tasks, whereas the evaluated reranking and LLM-based candidate-scoring methods provide little or no end-to-end improvement. Errors frequently reflect diseases matching only subsets of the phenotype evidence. These findings indicate that the evaluated embedding and reranking configurations do not reliably recover the compatibility encoded by the selected ontology relations and phenotype combinations and motivate retrieval systems that better integrate learned representations with structured biomedical knowledge.
Title:OntologyBench: Can Dense Retrieval Satisfy Structured Biomedical Constraints?
Abstract:We introduce OntologyBench, a tiered biomedical retrieval benchmark comprising 471,854 training and 125,744 evaluation query-document relevance pairs across concept grounding, relational retrieval, and compositional phenotype-based retrieval. Although these tasks can be tractable using ontology-aware reference methods, across task tiers, embedding performance is generally lower on relational and compositional tasks than on concept-grounding tasks. Fine-tuning on ontology-derived supervision improves performance on several relational and compositional tasks, whereas the evaluated reranking and LLM-based candidate-scoring methods provide little or no end-to-end improvement. Errors frequently reflect diseases matching only subsets of the phenotype evidence. These findings indicate that the evaluated embedding and reranking configurations do not reliably recover the compatibility encoded by the selected ontology relations and phenotype combinations and motivate retrieval systems that better integrate learned representations with structured biomedical knowledge.
19 pages and 1 figure. Accepted to EMNLP 2026 Main conference. this https URL
Artificial Intelligence (cs.AI)
arXiv:2609.08174 [cs.AI]
(or arXiv:2609.08174v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.08174 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 244 words
Related coverage
Co-adaptive Rule Routing and Constraining for Structured Knowledge Retrieval
arXiv · Oct 15, 2026
RareDx: Controlled Knowledge Integration and Graph-Grounded Policy Optimization for Rare-Disease Diagnosis
arXiv · Sep 29, 2026
Presentation: From Models to Agents: Building Context-Aware Consumer AI at Scale at DoorDash
InfoQ · Aug 15, 2026
Predictable Failure in Multi-Hop Retrieval: Score-Distributional Confidence Scoring and Abstention
arXiv · Sep 21, 2026