aiqbench analysis
This research moves beyond merely detecting hallucinations in Retrieval-Augmented Generation (RAG) systems by empirically comparing three repair strategies: deletion, replacement with source text, and rewriting. The study reveals that while all methods reduce unsupported content, they operate on a trade-off between factual grounding preservation and information retention, with no single strategy universally outperforming the others across different utility contexts. For developers building grounded AI assistants, these findings highlight the necessity of evidence-based selection over reliance on automatic metrics alone when deciding how to handle flagged claims.
Detecting and Repairing Hallucinations in Retrieval-Augmented Generation
arXiv:2608.29307v1 Announce Type: cross Abstract: Language models increasingly answer questions by consulting retrieved documents rather than memory alone, a design now common in search assistants and enterprise knowledge tools. Grounding a model in retrieved text reduces unsupported statements but does not eliminate them, and a reader cannot tell a grounded sentence from an invented one. Most research on this problem stops at detection, yet flagging a faulty answer changes nothing for the person reading it, and little is known about which action should follow. Using RAGTruth, a benchmark whose unsupported passages are annotated by hand, we split each flagged answer into individual factual claims, check each against the retrieved source, and compare leaving the answer untouched with three repair strategies of increasing richness: deleting an unsupported claim, replacing it with source text, and rewriting it. Three language models from different families judge the 916 repaired answers. Every strategy reduces the proportion of answers judged to contain unsupported content, and all three judges agree on the ordering. Deletion achieves the largest reduction while retaining least of the original answer, at 64.3% of the text, whereas rewriting retains 80.1% and reduces least. Repair is not confined to faulty answers: 83.5% of answers annotated clean are edited too. The strategies occupy different points on a grounding preservation trade-off rather than forming a quality ranking, and choosing between them needs evidence about answer usefulness that automatic metrics cannot supply.
Abstract:Language models increasingly answer questions by consulting retrieved documents rather than memory alone, a design now common in search assistants and enterprise knowledge tools. Grounding a model in retrieved text reduces unsupported statements but does not eliminate them, and a reader cannot tell a grounded sentence from an invented one. Most research on this problem stops at detection, yet flagging a faulty answer changes nothing for the person reading it, and little is known about which action should follow. Using RAGTruth, a benchmark whose unsupported passages are annotated by hand, we split each flagged answer into individual factual claims, check each against the retrieved source, and compare leaving the answer untouched with three repair strategies of increasing richness: deleting an unsupported claim, replacing it with source text, and rewriting it. Three language models from different families judge the 916 repaired answers. Every strategy reduces the proportion of answers judged to contain unsupported content, and all three judges agree on the ordering. Deletion achieves the largest reduction while retaining least of the original answer, at 64.3% of the text, whereas rewriting retains 80.1% and reduces least. Repair is not confined to faulty answers: 83.5% of answers annotated clean are edited too. The strategies occupy different points on a grounding preservation trade-off rather than forming a quality ranking, and choosing between them needs evidence about answer usefulness that automatic metrics cannot supply.
15 pages, 3 figures, 7 tables. Submitted to BNAIC/BeNeLearn 2026 as a Type A paper
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2608.29307 [cs.CL]
(or arXiv:2608.29307v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.29307 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 343 words
Related coverage
RAGAL: A Frugal, Fully Local Retrieval-Augmented Assistant for Technical Support at a Government Agency
arXiv · Jul 22, 2026
X-MADAM-RAG: Diagnosing and Handling Chinese-English Evidence Conflict in Retrieval-Augmented Generation
arXiv · Jun 12, 2026
Article: Your Next DSL Author is a Language Model
InfoQ · Sep 16, 2026
Evidence-Aligned Entity Verification for Hallucination Detection in Retrieval-Augmented Generation
arXiv · Sep 9, 2026