aiqbench analysis
RAGMark introduces a modular benchmarking suite specifically designed for small-scale multi-GPU environments, enabling detailed analysis of latency, energy consumption, and resource bottlenecks across various RAG components. By quantifying the compounding benefits of reranking and compression techniques, the framework provides critical insights into optimizing context reduction strategies to lower power usage and improve throughput in retrieval-augmented generation pipelines.
RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systems
arXiv:2609.05760v1 Announce Type: cross Abstract: We present RAGMark, a modular benchmarking framework for advanced Retrieval-Augmented Generation (RAG) systems targeting small-scale multi-GPU environments. RAGMark evaluates diverse RAG components, including retrievers, vector databases, prompt-processing methods, and generator models, while collecting detailed per-stage metrics such as latency, GPU utilization, memory consumption, power usage, time to first token (TTFT), throughput, and answer quality. The framework is highly extensible, separating RAG stages, timing, and resource monitoring into modular components, and is designed to efficiently sweep large configuration spaces while minimizing repeated model and database initialization overhead. Using RAGMark, we characterize five RAG workloads on open-domain QA datasets across varying retrieval depths, model scales, reranking, compression methods, and vector database configurations. We show that while autoregressive generation dominates latency in naive pipelines, context-reduction techniques shift bottlenecks across compute, memory bandwidth, and preprocessing stages. Reranking and compression produce compounding benefits: reranking reduces compression workload itself, while both jointly reduce prefill and KV-cache traversal costs, lowering energy consumption by up to 66%. We further observe strong cross-stage interactions, where small upstream context reductions cascade through downstream latency, memory traffic, and energy consumption. The RAGMark source code is publicly available at: https://github.com/zferic/RAGMark.
Abstract:We present RAGMark, a modular benchmarking framework for advanced Retrieval-Augmented Generation (RAG) systems targeting small-scale multi-GPU environments. RAGMark evaluates diverse RAG components, including retrievers, vector databases, prompt-processing methods, and generator models, while collecting detailed per-stage metrics such as latency, GPU utilization, memory consumption, power usage, time to first token (TTFT), throughput, and answer quality. The framework is highly extensible, separating RAG stages, timing, and resource monitoring into modular components, and is designed to efficiently sweep large configuration spaces while minimizing repeated model and database initialization overhead. Using RAGMark, we characterize five RAG workloads on open-domain QA datasets across varying retrieval depths, model scales, reranking, compression methods, and vector database configurations. We show that while autoregressive generation dominates latency in naive pipelines, context-reduction techniques shift bottlenecks across compute, memory bandwidth, and preprocessing stages. Reranking and compression produce compounding benefits: reranking reduces compression workload itself, while both jointly reduce prefill and KV-cache traversal costs, lowering energy consumption by up to 66%. We further observe strong cross-stage interactions, where small upstream context reductions cascade through downstream latency, memory traffic, and energy consumption. The RAGMark source code is publicly available at: this https URL.
Performance (cs.PF); Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR); Information Retrieval (cs.IR)
arXiv:2609.05760 [cs.PF]
(or arXiv:2609.05760v1 [cs.PF] for this version)
https://doi.org/10.48550/arXiv.2609.05760 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 292 words
Related coverage
Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model
AMD · Jul 24, 2026
Fine-grained Claim-level RAG Benchmark for Law
arXiv · Aug 18, 2026
MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval
arXiv · Jun 8, 2026
MITRE-SAGE: A Multi-Agent Cybersecurity Question-Answering Model
arXiv · Aug 20, 2026