analiza aiqbench
RAGStress introduces a rigorous benchmark for evaluating Retrieval-Augmented Generation systems under realistic knowledge base corruption, addressing a critical gap in current robustness testing. By systematically applying semantic-fidelity corruptions such as factual errors and contradiction injection, the study reveals that clean retrieval metrics often mask significant vulnerabilities in model behavior. This research provides essential tools for developers to stress-test RAG pipelines before deployment in production environments where data integrity cannot be guaranteed.
RAGStress: A controlled benchmark for evaluating retrieval-augmented generation under knowledge-base degradation
arXiv:2610.04691v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) is typically evaluated under the implicit assumption that the underlying knowledge base (KB) is clean, leaving the behaviour of RAG systems under realistic KB degradation poorly characterised. We introduce RAGStress, a controlled evaluation benchmark for stress-testing RAG systems under systematic KB corruption. The benchmark pairs four naturalistic corruption types (factual corruption, numeric typo, relevance poisoning, and contradiction injection) with three severity levels (subtle, moderate, and obvious) over a single-KB, metadata-filtered experimental design built from 57 MMLU subjects and 182,546 documents. Across 52,500 model-question-condition evaluations, RAGStress reveals that clean retrieval can mask robustness differences, semantic-fidelity corruptions are substantially more harmful than signal-utility perturbations, no-retrieval accuracy does not predict corrupted-retrieval robustness, and mixed-KB accuracy should not be treated as worst-case robustness. We document the benchmark's intended use, supported claims, and limitations, and provide an artifact bundle including generation scripts, corruption prompts, metadata schema, and evaluation code. RAGStress is intended as a controlled stress test for RAG robustness under KB corruption, not as a general model leaderboard.
Abstract:Retrieval-Augmented Generation (RAG) is typically evaluated under the implicit assumption that the underlying knowledge base (KB) is clean, leaving the behaviour of RAG systems under realistic KB degradation poorly characterised. We introduce RAGStress, a controlled evaluation benchmark for stress-testing RAG systems under systematic KB corruption. The benchmark pairs four naturalistic corruption types (factual corruption, numeric typo, relevance poisoning, and contradiction injection) with three severity levels (subtle, moderate, and obvious) over a single-KB, metadata-filtered experimental design built from 57 MMLU subjects and 182,546 documents. Across 52,500 model-question-condition evaluations, RAGStress reveals that clean retrieval can mask robustness differences, semantic-fidelity corruptions are substantially more harmful than signal-utility perturbations, no-retrieval accuracy does not predict corrupted-retrieval robustness, and mixed-KB accuracy should not be treated as worst-case robustness. We document the benchmark's intended use, supported claims, and limitations, and provide an artifact bundle including generation scripts, corruption prompts, metadata schema, and evaluation code. RAGStress is intended as a controlled stress test for RAG robustness under KB corruption, not as a general model leaderboard.
Artificial Intelligence (cs.AI)
arXiv:2610.04691 [cs.AI]
(or arXiv:2610.04691v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.04691 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 261 cuvinte
Subiecte conexe
Agentic Hybrid RAG for Evidence-Grounded Muon Collider Analysis
arXiv · 30 sept. 2026
Agentic Hybrid RAG for Evidence-Grounded Muon Collider Analysis
arXiv · 10 iun. 2026
Introducing Mistral OCR 4
Mistral AI · 23 iun. 2026
Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation
arXiv · 18 sept. 2026