aiqbench analysis
This research introduces EnComp, a lightweight encoder-only framework designed to compress context in retrieval-augmented generation without sacrificing accuracy. By leveraging counterfactual training signals and contrastive ranking objectives, the method significantly reduces memory usage and latency while preserving critical evidence for LLM inference. The approach offers a practical solution for deploying efficient question-answering systems in resource-constrained environments.
EnComp: Lightweight Encoder-Only Context Compression for Retrieval-Augmented Question Answering
arXiv:2603.09222v2 Announce Type: replace Abstract: Efficient context compression is critical for retrieval-augmented question answering in resource-constrained settings, where long retrieved contexts increase latency, memory use, and LLM reader cost. We propose a lightweight encoder-only framework for query-driven sentence pruning that preserves answer-critical evidence while aggressively reducing irrelevant context. Our method learns marginal contribution scores for sentences using counterfactual training signals and optimizes a contrastive ranking objective that separates critical evidence from noncritical context. Our approach scores all sentences from a single full-context encoding, enabling fast inference with low computational overhead. Experiments show that it maintains accuracy comparable to the strongest baseline while using 3.7$\times$ less peak memory and achieving nearly 3$\times$ lower compression latency, demonstrating an effective quality--efficiency trade-off for practical resource-constrained deployment.
Abstract:Efficient context compression is critical for retrieval-augmented question answering in resource-constrained settings, where long retrieved contexts increase latency, memory use, and LLM reader cost. We propose a lightweight encoder-only framework for query-driven sentence pruning that preserves answer-critical evidence while aggressively reducing irrelevant context. Our method learns marginal contribution scores for sentences using counterfactual training signals and optimizes a contrastive ranking objective that separates critical evidence from noncritical context. Our approach scores all sentences from a single full-context encoding, enabling fast inference with low computational overhead. Experiments show that it maintains accuracy comparable to the strongest baseline while using 3.7$\times$ less peak memory and achieving nearly 3$\times$ lower compression latency, demonstrating an effective quality--efficiency trade-off for practical resource-constrained deployment.
Computation and Language (cs.CL)
arXiv:2603.09222 [cs.CL]
(or arXiv:2603.09222v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2603.09222 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 211 words
Related coverage
Rethinking LoRA Memory Through the Lens of KV Cache Compression
arXiv · Jun 5, 2026
Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression
arXiv · Jun 15, 2026
AdaMem: Adaptive Memory Token Allocation for Soft Compression in Retrieval-Augmented Generation
arXiv · Sep 22, 2026
ARC-Encoder: learning compressed text representations for large language models
arXiv · Jul 31, 2026