aiqbench analysis
This research evaluates context compaction strategies in language-model agents, revealing that methods optimizing for immediate questions often fail to preserve evidence needed for subsequent queries. The study demonstrates that random deletion can sometimes outperform active sentence selection on later tasks, while word-level compression like LLMLingua-2 offers a more robust alternative by retaining critical content words. These findings highlight the inherent trade-offs in context management when agents must balance current task performance with future query reliability.
Evaluating Context Compaction on the Questions That Come Later
arXiv:2610.27301v1 Announce Type: new Abstract: Language-model agents compact their context for the question at hand, but the compacted context must also serve questions that come later. We evaluate compaction on these later questions, using four corpora that annotate the evidence of several questions per text and recorded agent trajectories, and compare every method with random deletion at the same budget. On MultiRC, keeping the sentences that gpt-6-luna (through the OpenAI Decisions API) judges necessary for the current question raises its accuracy on that question by 11.4 points at 25% of the tokens. On the later questions, however, no method that selects sentences for the current question does detectably better than deleting sentences at random. Deleting words instead of sentences does better: LLMLingua-2, which ignores the question, beats random sentence deletion at the same token count by 12.5 points on the later MultiRC questions. It keeps about twice random deletion's share of the evidence's content words, and at equal content-word retention its gain shrinks to 7.3 points (exploratory). On FEVER, label accuracy can overstate what the kept text supports: the gain of sentence selection is no longer detected when answers are scored against the kept evidence, and the 21.0-point gain of LLMLingua-2 is not shown to rest on the kept text. The ranking also depends on the setting: on Evidence Inference sentence selection beats LLMLingua-2 at a nominal 25% budget, and on $\tau$-bench trajectories, whose later tool calls need exact identifiers, it beats random deletion while LLMLingua-2 shows no detected gain. Scores on the question a compaction was made for do not reliably show how it serves later ones.
Abstract:Language-model agents compact their context for the question at hand, but the compacted context must also serve questions that come later. We evaluate compaction on these later questions, using four corpora that annotate the evidence of several questions per text and recorded agent trajectories, and compare every method with random deletion at the same budget. On MultiRC, keeping the sentences that gpt-6-luna (through the OpenAI Decisions API) judges necessary for the current question raises its accuracy on that question by 11.4 points at 25% of the tokens. On the later questions, however, no method that selects sentences for the current question does detectably better than deleting sentences at random. Deleting words instead of sentences does better: LLMLingua-2, which ignores the question, beats random sentence deletion at the same token count by 12.5 points on the later MultiRC questions. It keeps about twice random deletion's share of the evidence's content words, and at equal content-word retention its gain shrinks to 7.3 points (exploratory). On FEVER, label accuracy can overstate what the kept text supports: the gain of sentence selection is no longer detected when answers are scored against the kept evidence, and the 21.0-point gain of LLMLingua-2 is not shown to rest on the kept text. The ranking also depends on the setting: on Evidence Inference sentence selection beats LLMLingua-2 at a nominal 25% budget, and on $\tau$-bench trajectories, whose later tool calls need exact identifiers, it beats random deletion while LLMLingua-2 shows no detected gain. Scores on the question a compaction was made for do not reliably show how it serves later ones.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2610.27301 [cs.CL]
(or arXiv:2610.27301v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.27301 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
macros/compress_fever.tex
macros/compress_fever_tfidf.tex
macros/compress_third.tex
macros/compress_third_tfidf.tex
macros/locomo.tex
macros/locomo_hard.tex
macros/numbers3.tex
macros/p4_audit.tex
macros/p4_cost.tex
macros/p4_extra.tex
macros/p4_fever_support.tex
macros/p4_granularity.tex
macros/p4_hybrid.tex
macros/p4_known.tex
macros/p4_overlap.tex
macros/p4_reader.tex
macros/p4_refusal.tex
macros/p4_retention.tex
macros/p4_retention_labels.tex
macros/p4_sensitivity.tex
macros/p4_summary.tex
macros/p4_tau.tex
macros/p4_whywords.tex
macros/p4new.tex
macros/restore.tex
results/compress.csv
results/compress_ci_oai-luna.csv
results/compress_ci_open-bosun.csv
results/compress_ci_open-decider.csv
results/compress_ci_open-lumma.csv
results/compress_fever_ci.csv
results/compress_fever_gate.csv
results/compress_fever_main.csv
results/compress_fever_rationale.csv
results/compress_fever_tfidf.csv
results/compress_gate_oai-luna.csv
results/compress_gate_open-bosun.csv
results/compress_gate_open-decider.csv
results/compress_gate_open-lumma.csv
results/compress_oai-luna.csv
results/compress_open-bosun.csv
results/compress_open-decider.csv
results/compress_open-lumma.csv
results/compress_rationale_link_oai-luna.csv
results/compress_rationale_link_open-bosun.csv
results/compress_rationale_link_open-decider.csv
results/compress_rationale_link_open-lumma.csv
results/compress_rationale_oai-luna.csv
results/compress_rationale_open-bosun.csv
results/compress_rationale_open-decider.csv
results/compress_rationale_open-lumma.csv
results/compress_third_ci.csv
results/compress_third_info.csv
results/compress_third_main.csv
results/compress_third_tfidf_ci.csv
results/compress_third_tfidf_main.csv
results/locomo_acc.csv
results/locomo_ci.csv
results/locomo_hard.csv
results/locomo_hypotheses.csv
results/locomo_keepdrop.csv
results/locomo_stats.csv
results/locomo_tokens.csv
results/p4_audit_floor.csv
results/p4_audit_groups.csv
results/p4_audit_label.csv
results/p4_audit_readers.csv
results/p4_cost_amortized.csv
results/p4_cost_components.csv
results/p4_cost_keepdrop.csv
results/p4_cost_model.csv
results/p4_cost_rule.csv
results/p4_cost_tokshare.csv
results/p4_em.csv
results/p4_extra.csv
results/p4_fever_support_contrasts.csv
results/p4_fever_support_levels.csv
results/p4_granularity_contrasts.csv
results/p4_granularity_coverage.csv
results/p4_granularity_levels.csv
results/p4_hybrid_contrasts.csv
results/p4_hybrid_coverage.csv
results/p4_hybrid_levels.csv
results/p4_hybrid_predictions.csv
results/p4_interaction.csv
results/p4_known_contrasts.csv
results/p4_known_k4.csv
results/p4_known_line.csv
results/p4_known_predictions.csv
results/p4_overlap.csv
results/p4_reader_contrasts.csv
results/p4_reader_gate.csv
results/p4_reader_levels.csv
results/p4_refusal_contrasts.csv
results/p4_refusal_levels.csv
results/p4_refusal_rates.csv
results/p4_retention.csv
results/p4_retention_labels.csv
results/p4_sensitivity_balanced.csv
results/p4_sensitivity_loco.csv
results/p4_summary_contrasts.csv
results/p4_summary_levels.csv
results/p4_summary_predictions.csv
results/p4_summary_tokens.csv
results/p4_tau_contrasts.csv
results/p4_tau_gap_closed.csv
results/p4_tau_recoverable.csv
results/p4_tau_refusals.csv
results/p4_tau_retention.csv
results/p4_whywords_features.csv
results/p4_whywords_gaps.csv
results/p4fu_ab.csv
results/p4fu_ab_levels.csv
results/p4fu_c.csv
results/p4fu_c_levels.csv
results/p4fu_d.csv
results/p4fu_d_levels.csv
results/p4new_drift.csv
results/p4new_e1_per_mask.csv
results/p4new_e1_random_masks.csv
results/p4new_e2_lingua.csv
results/p4new_e2_retention.csv
results/p4new_e3_keepdrop.csv
results/restore_conc_oai-luna.csv
results/restore_dose_oai-luna.csv
results/restore_oai-luna.csv
(124 additional files not shown) You must enabled JavaScript to view entire file list.
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 501 words
Related coverage
Lost in Context Compaction: When LLM Systems Drop Side Constraints
arXiv · Oct 15, 2026
MAVEN: A Macro-Societal Value Evaluation Framework of Multimodal Content with Compact Aligned Evaluators
arXiv · Aug 20, 2026
Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models
arXiv · Sep 14, 2026
Judging in Latent Space: Efficient Generative Reward Modeling via Semantics-Preserving Compression
arXiv · Oct 8, 2026