aiqbench analysis
This research demonstrates that for long-document retrieval, the structural efficiency of a deterministic segment tree can outperform language model-generated summaries when evaluated at matched computational costs. By eliminating LLM calls during indexing and using the tree solely as a navigation scaffold, NavTree achieves superior recall in multi-hop QA tasks compared to both flat retrievers and standard hierarchical methods like RAPTOR. This finding shifts the focus from content generation to architectural optimization for scaling retrieval-augmented generation systems.
Tree Navigation Without LLM Summaries: A Matched-Cost Study of Hierarchical Retrieval for Long-Document QA
arXiv:2610.06902v1 Announce Type: new Abstract: Retrieval-augmented generation grounds language models in external context, but for long documents flat top-$k$ retrieval can cluster on a single region and miss complementary evidence. RAPTOR-style summary trees address this by recursively clustering chunks and using a language model to summarize each cluster at indexing time, then ranking summary nodes alongside raw chunks at query time. We show the main benefit of summary trees in long-document QA can come from navigation rather than the generated summary content. We introduce NavTree, a leaves-only retriever that builds a deterministic balanced segment tree over chunks (zero language-model calls at indexing) and uses the tree purely as a navigation scaffold: a hybrid lexical-and-dense frontier walk, anchored on top retrieved leaves, descends from the root and emits only leaf chunks to the reader. On a matched-cost evaluation against flat retrievers and an extractive re-implementation of RAPTOR, NavTree is the strongest matched-cost hierarchical retriever in our evaluated grid and ties the strongest flat baseline. On long-document multi-hop QA, it is the only hierarchical method that significantly beats BM25 on a class-vs-class basis, corroborated by a reader-free retrieval-recall check. A matched-reader replication of the published abstractive RAPTOR variant, given strong cluster summaries, still loses to NavTree at every multi-chunk budget, at zero indexing cost. The ranking carries across stronger and open-weight readers, a stronger encoder, and a full factorial that isolates leaves-only emission as the structural lever.
Abstract:Retrieval-augmented generation grounds language models in external context, but for long documents flat top-$k$ retrieval can cluster on a single region and miss complementary evidence. RAPTOR-style summary trees address this by recursively clustering chunks and using a language model to summarize each cluster at indexing time, then ranking summary nodes alongside raw chunks at query time. We show the main benefit of summary trees in long-document QA can come from navigation rather than the generated summary content. We introduce NavTree, a leaves-only retriever that builds a deterministic balanced segment tree over chunks (zero language-model calls at indexing) and uses the tree purely as a navigation scaffold: a hybrid lexical-and-dense frontier walk, anchored on top retrieved leaves, descends from the root and emits only leaf chunks to the reader. On a matched-cost evaluation against flat retrievers and an extractive re-implementation of RAPTOR, NavTree is the strongest matched-cost hierarchical retriever in our evaluated grid and ties the strongest flat baseline. On long-document multi-hop QA, it is the only hierarchical method that significantly beats BM25 on a class-vs-class basis, corroborated by a reader-free retrieval-recall check. A matched-reader replication of the published abstractive RAPTOR variant, given strong cluster summaries, still loses to NavTree at every multi-chunk budget, at zero indexing cost. The ranking carries across stronger and open-weight readers, a stronger encoder, and a full factorial that isolates leaves-only emission as the structural lever.
Computation and Language (cs.CL); Information Retrieval (cs.IR)
arXiv:2610.06902 [cs.CL]
(or arXiv:2610.06902v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.06902 Focus to learn more arXiv-issued DOI via DataCite
REPRODUCIBILITY.md
code/asrmb/__init__.py
code/asrmb/baselines/__init__.py
code/asrmb/baselines/bm25.py
code/asrmb/baselines/dense_flat.py
code/asrmb/baselines/hybrid_flat.py
code/asrmb/baselines/raptor_baseline.py
code/asrmb/benchmarks/__init__.py
code/asrmb/benchmarks/base.py
code/asrmb/benchmarks/multihop.py
code/asrmb/benchmarks/narrativeqa.py
code/asrmb/benchmarks/qasper.py
code/asrmb/benchmarks/quality.py
code/asrmb/core/__init__.py
code/asrmb/core/inference.py
code/asrmb/core/packing.py
code/asrmb/core/segtree.py
code/asrmb/index/__init__.py
code/asrmb/index/faiss_utils.py
code/asrmb/retrieval/__init__.py
code/asrmb/retrieval/ce_rerank.py
code/asrmb/retrieval/navigated_fill.py
code/requirements.txt
code/scripts/bench_index_cost.py
code/scripts/bench_query_latency.py
code/scripts/build_anon_zip.sh
code/scripts/build_paper_figures.py
code/scripts/build_paper_tables.py
code/scripts/compute_confidence_intervals.py
code/scripts/diagnostics/qasper_tail_token_stats.py
code/scripts/exp_abstractive_raptor.py
code/scripts/exp_alpha_kleaf_sweep.sh
code/scripts/exp_factorial_w7_quality2000.sh
code/scripts/exp_hybrid_flat_multihop_fill.sh
code/scripts/exp_multi_seed.sh
code/scripts/exp_multihop.sh
code/scripts/exp_queue_remaining.sh
code/scripts/exp_summary_vs_leaf.py
code/scripts/exp_summary_vs_leaf_extractive.py
code/scripts/exp_winning_cell_ablation.sh
code/scripts/merge_hybrid_flat_multihop.py
code/scripts/paper_status.sh
code/scripts/phase1_build_contexts.py
code/scripts/phase1_longdoc.py
code/scripts/phase2_longdoc.py
code/scripts/phase2_run_reader.py
code/scripts/plot_results.py
code/scripts/post_qwen_qasper_finalize.sh
code/scripts/render_appendix_table.py
code/scripts/render_factorial_w7.py
code/scripts/render_table_multihop_recall.py
code/scripts/render_table_openweight.py
code/scripts/render_table_qasper_recall.py
code/scripts/rescore_nqa_summary_leaf.py
code/scripts/resume_arr_experiments.sh
code/scripts/run_a1_ollama.sh
code/scripts/run_a1_openweight.sh
code/scripts/run_a3_bge_large.sh
code/scripts/run_ablation_qasper256.sh
code/scripts/run_benchmarks.py
code/scripts/run_canonical_baseline.sh
code/scripts/run_multihop_recall.py
code/scripts/run_niah.py
code/scripts/run_niah_multi.py
code/scripts/run_overnight_orchestrator.sh
code/scripts/run_paired_tests.py
code/scripts/run_paired_tests_openweight.py
code/scripts/run_qasper_recall.py
code/scripts/sync_asrmb_paper_tables.py
code/scripts/sync_quality_asrmb_tables_from_canon.py
responsible_nlp_checklist.md
results/abstractive_raptor_gpt4o/quality_checkpoint.json
results/abstractive_raptor_gpt4o/quality_predictions.json
results/abstractive_raptor_gpt4o/quality_results.json
results/abstractive_raptor_gpt4o/summary.json
results/bge_large_run/qasper/qasper_checkpoint.json
results/bge_large_run/qasper/qasper_results.csv
results/bge_large_run/qasper/qasper_results.json
results/bge_large_run/quality/quality_checkpoint.json
results/bge_large_run/quality/quality_results.csv
results/bge_large_run/quality/quality_results.json
results/canonical_v1/narrativeqa/narrativeqa_checkpoint.json
results/canonical_v1/narrativeqa/narrativeqa_results.csv
results/canonical_v1/narrativeqa/narrativeqa_results.json
results/canonical_v1/qasper/qasper_checkpoint.json
results/canonical_v1/qasper/qasper_results.csv
results/canonical_v1/qasper/qasper_results.json
results/canonical_v1/quality/quality_checkpoint.json
results/canonical_v1/quality/quality_results.csv
results/canonical_v1/quality/quality_results.json
results/factorial_w7_quality2000/A0_B0_C0_D0/quality_checkpoint.json
results/factorial_w7_quality2000/A0_B0_C0_D0/quality_results.csv
results/factorial_w7_quality2000/A0_B0_C0_D0/quality_results.json
results/factorial_w7_quality2000/A0_B0_C0_D1/quality_checkpoint.json
results/factorial_w7_quality2000/A0_B0_C0_D1/quality_results.csv
results/factorial_w7_quality2000/A0_B0_C0_D1/quality_results.json
results/factorial_w7_quality2000/A0_B0_C1_D0/quality_checkpoint.json
results/factorial_w7_quality2000/A0_B0_C1_D0/quality_results.csv
results/factorial_w7_quality2000/A0_B0_C1_D0/quality_results.json
results/factorial_w7_quality2000/A0_B0_C1_D1/quality_checkpoint.json
results/factorial_w7_quality2000/A0_B0_C1_D1/quality_results.csv
results/factorial_w7_quality2000/A0_B0_C1_D1/quality_results.json
results/factorial_w7_quality2000/A0_B1_C0_D0/quality_checkpoint.json
results/factorial_w7_quality2000/A0_B1_C0_D0/quality_results.csv
results/factorial_w7_quality2000/A0_B1_C0_D0/quality_results.json
results/factorial_w7_quality2000/A0_B1_C0_D1/quality_checkpoint.json
results/factorial_w7_quality2000/A0_B1_C0_D1/quality_results.csv
results/factorial_w7_quality2000/A0_B1_C0_D1/quality_results.json
results/factorial_w7_quality2000/A0_B1_C1_D0/quality_checkpoint.json
results/factorial_w7_quality2000/A0_B1_C1_D0/quality_results.csv
results/factorial_w7_quality2000/A0_B1_C1_D0/quality_results.json
results/factorial_w7_quality2000/A0_B1_C1_D1/quality_checkpoint.json
results/factorial_w7_quality2000/A0_B1_C1_D1/quality_results.csv
results/factorial_w7_quality2000/A0_B1_C1_D1/quality_results.json
results/factorial_w7_quality2000/A1_B0_C0_D0/quality_checkpoint.json
results/factorial_w7_quality2000/A1_B0_C0_D0/quality_results.csv
results/factorial_w7_quality2000/A1_B0_C0_D0/quality_results.json
results/factorial_w7_quality2000/A1_B0_C0_D1/quality_checkpoint.json
results/factorial_w7_quality2000/A1_B0_C0_D1/quality_results.csv
results/factorial_w7_quality2000/A1_B0_C0_D1/quality_results.json
results/factorial_w7_quality2000/A1_B0_C1_D0/quality_checkpoint.json
results/factorial_w7_quality2000/A1_B0_C1_D0/quality_results.csv
results/factorial_w7_quality2000/A1_B0_C1_D0/quality_results.json
results/factorial_w7_quality2000/A1_B0_C1_D1/quality_checkpoint.json
results/factorial_w7_quality2000/A1_B0_C1_D1/quality_results.csv
results/factorial_w7_quality2000/A1_B0_C1_D1/quality_results.json
results/factorial_w7_quality2000/A1_B1_C0_D0/quality_checkpoint.json
results/factorial_w7_quality2000/A1_B1_C0_D0/quality_results.csv
results/factorial_w7_quality2000/A1_B1_C0_D0/quality_results.json
results/factorial_w7_quality2000/A1_B1_C0_D1/quality_checkpoint.json
results/factorial_w7_quality2000/A1_B1_C0_D1/quality_results.csv
results/factorial_w7_quality2000/A1_B1_C0_D1/quality_results.json
results/factorial_w7_quality2000/A1_B1_C1_D0/quality_checkpoint.json
results/factorial_w7_quality2000/A1_B1_C1_D0/quality_results.csv
results/factorial_w7_quality2000/A1_B1_C1_D0/quality_results.json
results/factorial_w7_quality2000/A1_B1_C1_D1/quality_checkpoint.json
results/factorial_w7_quality2000/A1_B1_C1_D1/quality_results.csv
results/factorial_w7_quality2000/A1_B1_C1_D1/quality_results.json
results/factorial_w7_quality2000/_summary.json
results/final_archive/README.md
results/final_archive/longdoc_ci.json
results/final_archive/longdoc_results.json
results/final_archive/longdoc_results.json.checkpoint
results/final_archive/narrativeqa_checkpoint.json
results/final_archive/narrativeqa_results.json
results/final_archive/qasper_checkpoint.json
results/final_archive/qasper_ci.json
results/final_archive/qasper_results.json
results/final_archive/qasper_unifiedqa_results.json
results/final_archive/quality_ci.json
results/final_archive/quality_navigated_checkpoint.json
results/final_archive/quality_results.json
results/final_archive/quality_unifiedqa_results.json
results/final_archive/quality_unifiedqa_results.json.checkpoint
results/frozen_opt/AUDIT.md
results/frozen_opt/ablation_qasper256/_meta/manifest.json
results/frozen_opt/ablation_qasper256/row1_classic_steiner/qasper_checkpoint.json
results/frozen_opt/ablation_qasper256/row1_classic_steiner/qasper_ci.json
results/frozen_opt/ablation_qasper256/row1_classic_steiner/qasper_results.csv
results/frozen_opt/ablation_qasper256/row1_classic_steiner/qasper_results.json
results/frozen_opt/ablation_qasper256/row2_kleaf48/qasper_checkpoint.json
results/frozen_opt/ablation_qasper256/row2_kleaf48/qasper_ci.json
results/frozen_opt/ablation_qasper256/row2_kleaf48/qasper_results.csv
results/frozen_opt/ablation_qasper256/row2_kleaf48/qasper_results.json
results/frozen_opt/ablation_qasper256/row3_hybrid_alpha035/qasper_checkpoint.json
results/frozen_opt/ablation_qasper256/row3_hybrid_alpha035/qasper_ci.json
results/frozen_opt/ablation_qasper256/row3_hybrid_alpha035/qasper_results.csv
results/frozen_opt/ablation_qasper256/row3_hybrid_alpha035/qasper_results.json
results/frozen_opt/ablation_qasper256/row4_alpha050/qasper_checkpoint.json
results/frozen_opt/ablation_qasper256/row4_alpha050/qasper_ci.json
results/frozen_opt/ablation_qasper256/row4_alpha050/qasper_results.csv
results/frozen_opt/ablation_qasper256/row4_alpha050/qasper_results.json
results/frozen_opt/ablation_qasper256/row5_tight_tail/qasper_checkpoint.json
results/frozen_opt/ablation_qasper256/row5_tight_tail/qasper_ci.json
results/frozen_opt/ablation_qasper256/row5_tight_tail/qasper_results.csv
results/frozen_opt/ablation_qasper256/row5_tight_tail/qasper_results.json
results/frozen_opt/ablation_qasper256/summary.md
results/frozen_opt/ablation_quality2000/row1_classic_steiner/quality_checkpoint.json
results/frozen_opt/ablation_quality2000/row1_classic_steiner/quality_results.csv
results/frozen_opt/ablation_quality2000/row1_classic_steiner/quality_results.json
results/frozen_opt/ablation_quality2000/row2_kleaf48/quality_checkpoint.json
results/frozen_opt/ablation_quality2000/row2_kleaf48/quality_results.csv
results/frozen_opt/ablation_quality2000/row2_kleaf48/quality_results.json
results/frozen_opt/ablation_quality2000/row3_hybrid_alpha035/quality_checkpoint.json
results/frozen_opt/ablation_quality2000/row3_hybrid_alpha035/quality_results.csv
results/frozen_opt/ablation_quality2000/row3_hybrid_alpha035/quality_results.json
results/frozen_opt/ablation_quality2000/row4_alpha050/quality_checkpoint.json
results/frozen_opt/ablation_quality2000/row4_alpha050/quality_results.csv
results/frozen_opt/ablation_quality2000/row4_alpha050/quality_results.json
results/frozen_opt/ablation_quality2000/row5_full/quality_checkpoint.json
results/frozen_opt/ablation_quality2000/row5_full/quality_results.csv
results/frozen_opt/ablation_quality2000/row5_full/quality_results.json
results/frozen_opt/abstractive_raptor/narrativeqa_results.json
results/frozen_opt/abstractive_raptor/qasper_results.json
results/frozen_opt/abstractive_raptor/quality_results.json
results/frozen_opt/abstractive_raptor/summary.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.00_kleaf10/quality_checkpoint.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.00_kleaf10/quality_results.csv
results/frozen_opt/alpha_kleaf_sweep/alpha0.00_kleaf10/quality_results.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.25_kleaf10/quality_checkpoint.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.25_kleaf10/quality_results.csv
results/frozen_opt/alpha_kleaf_sweep/alpha0.25_kleaf10/quality_results.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.35_kleaf10/quality_checkpoint.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.35_kleaf10/quality_results.csv
results/frozen_opt/alpha_kleaf_sweep/alpha0.35_kleaf10/quality_results.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.35_kleaf24/quality_checkpoint.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.35_kleaf24/quality_results.csv
results/frozen_opt/alpha_kleaf_sweep/alpha0.35_kleaf24/quality_results.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.35_kleaf48/quality_checkpoint.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.35_kleaf48/quality_results.csv
results/frozen_opt/alpha_kleaf_sweep/alpha0.35_kleaf48/quality_results.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.35_kleaf5/quality_checkpoint.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.35_kleaf5/quality_results.csv
results/frozen_opt/alpha_kleaf_sweep/alpha0.35_kleaf5/quality_results.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.50_kleaf10/quality_checkpoint.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.50_kleaf10/quality_results.csv
results/frozen_opt/alpha_kleaf_sweep/alpha0.50_kleaf10/quality_results.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.65_kleaf10/quality_checkpoint.json
results/frozen_opt/alpha_kleaf_sweep/alpha0.65_kleaf10/quality_results.csv
results/frozen_opt/alpha_kleaf_sweep/alpha0.65_kleaf10/quality_results.json
results/frozen_opt/alpha_kleaf_sweep/alpha1.00_kleaf10/quality_checkpoint.json
results/frozen_opt/alpha_kleaf_sweep/alpha1.00_kleaf10/quality_results.csv
results/frozen_opt/alpha_kleaf_sweep/alpha1.00_kleaf10/quality_results.json
results/frozen_opt/canonical_v1/_meta/code_snapshot/CHECKSUMS.txt
results/frozen_opt/canonical_v1/_meta/code_snapshot/SNAPSHOT.txt
results/frozen_opt/canonical_v1/_meta/code_snapshot/bm25.py
results/frozen_opt/canonical_v1/_meta/code_snapshot/ce_rerank.py
results/frozen_opt/canonical_v1/_meta/code_snapshot/faiss_utils.py
results/frozen_opt/canonical_v1/_meta/code_snapshot/inference.py
results/frozen_opt/canonical_v1/_meta/code_snapshot/navigated_fill.py
results/frozen_opt/canonical_v1/_meta/code_snapshot/packing.py
results/frozen_opt/canonical_v1/_meta/code_snapshot/raptor_baseline.py
results/frozen_opt/canonical_v1/_meta/code_snapshot/run_benchmarks.py
results/frozen_opt/canonical_v1/_meta/code_snapshot/segtree.py
results/frozen_opt/canonical_v1/_meta/manifest.json
results/frozen_opt/canonical_v1/narrativeqa/narrativeqa_ci.json
results/frozen_opt/canonical_v1/narrativeqa/narrativeqa_ci_paired.json
results/frozen_opt/canonical_v1/narrativeqa/narrativeqa_predictions.json
results/frozen_opt/canonical_v1/narrativeqa/narrativeqa_predictions_raw.json
results/frozen_opt/canonical_v1/narrativeqa/narrativeqa_results.csv
results/frozen_opt/canonical_v1/narrativeqa/narrativeqa_results.json
results/frozen_opt/canonical_v1/qasper/qasper_ci.json
results/frozen_opt/canonical_v1/qasper/qasper_ci_paired.json
results/frozen_opt/canonical_v1/qasper/qasper_predictions.json
results/frozen_opt/canonical_v1/qasper/qasper_predictions_raw.json
results/frozen_opt/canonical_v1/qasper/qasper_results.csv
results/frozen_opt/canonical_v1/qasper/qasper_results.json
results/frozen_opt/canonical_v1/quality/quality_ci.json
results/frozen_opt/canonical_v1/quality/quality_ci_paired.json
results/frozen_opt/canonical_v1/quality/quality_predictions.json
results/frozen_opt/canonical_v1/quality/quality_predictions_raw.json
results/frozen_opt/canonical_v1/quality/quality_results.csv
results/frozen_opt/canonical_v1/quality/quality_results.json
results/frozen_opt/index_cost_bench/_meta/manifest.json
results/frozen_opt/index_cost_bench/per_doc.json
results/frozen_opt/index_cost_bench/summary.json
results/frozen_opt/index_cost_bench/summary.md
results/frozen_opt/multi_seed_quality1024/seed1/quality_checkpoint.json
results/frozen_opt/multi_seed_quality1024/seed1/quality_results.csv
results/frozen_opt/multi_seed_quality1024/seed1/quality_results.json
results/frozen_opt/multi_seed_quality1024/seed2/quality_results.json
results/frozen_opt/multi_seed_quality1024/seed2_rerun/quality_checkpoint.json
results/frozen_opt/multi_seed_quality1024/seed2_rerun/quality_results.csv
results/frozen_opt/multi_seed_quality1024/seed2_rerun/quality_results.json
results/frozen_opt/multi_seed_quality1024/seed3/quality_results.json
results/frozen_opt/multi_seed_quality1024/seed3_rerun/quality_checkpoint.json
results/frozen_opt/multi_seed_quality1024/seed3_rerun/quality_results.csv
results/frozen_opt/multi_seed_quality1024/seed3_rerun/quality_results.json
results/frozen_opt/multi_seed_quality1024/seed4/quality_results.json
results/frozen_opt/multi_seed_quality1024/seed4_rerun/quality_checkpoint.json
results/frozen_opt/multi_seed_quality1024/seed4_rerun/quality_results.csv
results/frozen_opt/multi_seed_quality1024/seed4_rerun/quality_results.json
results/frozen_opt/multi_seed_quality1024/seed5/quality_results.json
results/frozen_opt/multi_seed_quality1024/seed5_rerun/quality_checkpoint.json
results/frozen_opt/multi_seed_quality1024/seed5_rerun/quality_results.csv
results/frozen_opt/multi_seed_quality1024/seed5_rerun/quality_results.json
results/frozen_opt/nqa_alpha050/_meta/code_snapshot/CHECKSUMS.txt
results/frozen_opt/nqa_alpha050/_meta/code_snapshot/SNAPSHOT.txt
results/frozen_opt/nqa_alpha050/_meta/code_snapshot/bm25.py
results/frozen_opt/nqa_alpha050/_meta/code_snapshot/ce_rerank.py
results/frozen_opt/nqa_alpha050/_meta/code_snapshot/faiss_utils.py
results/frozen_opt/nqa_alpha050/_meta/code_snapshot/inference.py
results/frozen_opt/nqa_alpha050/_meta/code_snapshot/navigated_fill.py
results/frozen_opt/nqa_alpha050/_meta/code_snapshot/packing.py
results/frozen_opt/nqa_alpha050/_meta/code_snapshot/raptor_baseline.py
results/frozen_opt/nqa_alpha050/_meta/code_snapshot/run_benchmarks.py
results/frozen_opt/nqa_alpha050/_meta/code_snapshot/segtree.py
results/frozen_opt/nqa_alpha050/_meta/manifest.json
results/frozen_opt/nqa_alpha050/narrativeqa_cells.json
results/frozen_opt/nqa_alpha050/narrativeqa_ci.json
results/frozen_opt/nqa_alpha050/narrativeqa_predictions.json
results/frozen_opt/nqa_alpha050/narrativeqa_predictions_raw.json
results/frozen_opt/nqa_alpha050/narrativeqa_results.csv
results/frozen_opt/nqa_alpha050/narrativeqa_results.json
results/frozen_opt/nqa_short_prompt/_meta/code_snapshot/CHECKSUMS.txt
results/frozen_opt/nqa_short_prompt/_meta/code_snapshot/SNAPSHOT.txt
results/frozen_opt/nqa_short_prompt/_meta/code_snapshot/bm25.py
results/frozen_opt/nqa_short_prompt/_meta/code_snapshot/ce_rerank.py
results/frozen_opt/nqa_short_prompt/_meta/code_snapshot/faiss_utils.py
results/frozen_opt/nqa_short_prompt/_meta/code_snapshot/inference.py
results/frozen_opt/nqa_short_prompt/_meta/code_snapshot/navigated_fill.py
results/frozen_opt/nqa_short_prompt/_meta/code_snapshot/packing.py
results/frozen_opt/nqa_short_prompt/_meta/code_snapshot/raptor_baseline.py
results/frozen_opt/nqa_short_prompt/_meta/code_snapshot/run_benchmarks.py
results/frozen_opt/nqa_short_prompt/_meta/code_snapshot/segtree.py
results/frozen_opt/nqa_short_prompt/_meta/manifest.json
results/frozen_opt/nqa_short_prompt/narrativeqa_cells.json
results/frozen_opt/nqa_short_prompt/narrativeqa_ci.json
results/frozen_opt/nqa_short_prompt/narrativeqa_predictions.json
results/frozen_opt/nqa_short_prompt/narrativeqa_predictions_raw.json
results/frozen_opt/nqa_short_prompt/narrativeqa_results.csv
results/frozen_opt/nqa_short_prompt/narrativeqa_results.json
results/frozen_opt/qasper_ce/_meta/code_snapshot/CHECKSUMS.txt
results/frozen_opt/qasper_ce/_meta/code_snapshot/SNAPSHOT.txt
results/frozen_opt/qasper_ce/_meta/code_snapshot/bm25.py
results/frozen_opt/qasper_ce/_meta/code_snapshot/ce_rerank.py
results/frozen_opt/qasper_ce/_meta/code_snapshot/faiss_utils.py
results/frozen_opt/qasper_ce/_meta/code_snapshot/inference.py
results/frozen_opt/qasper_ce/_meta/code_snapshot/navigated_fill.py
results/frozen_opt/qasper_ce/_meta/code_snapshot/packing.py
results/frozen_opt/qasper_ce/_meta/code_snapshot/raptor_baseline.py
results/frozen_opt/qasper_ce/_meta/code_snapshot/run_benchmarks.py
results/frozen_opt/qasper_ce/_meta/code_snapshot/segtree.py
results/frozen_opt/qasper_ce/_meta/manifest.json
results/frozen_opt/qasper_ce/qasper_cells.json
results/frozen_opt/qasper_ce/qasper_ci.json
results/frozen_opt/qasper_ce/qasper_predictions.json
results/frozen_opt/qasper_ce/qasper_predictions_raw.json
results/frozen_opt/qasper_ce/qasper_results.csv
results/frozen_opt/qasper_ce/qasper_results.json
results/frozen_opt/quality_gpt4o_sota/_meta/code_snapshot/CHECKSUMS.txt
results/frozen_opt/quality_gpt4o_sota/_meta/code_snapshot/SNAPSHOT.txt
results/frozen_opt/quality_gpt4o_sota/_meta/code_snapshot/bm25.py
results/frozen_opt/quality_gpt4o_sota/_meta/code_snapshot/ce_rerank.py
results/frozen_opt/quality_gpt4o_sota/_meta/code_snapshot/faiss_utils.py
results/frozen_opt/quality_gpt4o_sota/_meta/code_snapshot/inference.py
results/frozen_opt/quality_gpt4o_sota/_meta/code_snapshot/navigated_fill.py
results/frozen_opt/quality_gpt4o_sota/_meta/code_snapshot/packing.py
results/frozen_opt/quality_gpt4o_sota/_meta/code_snapshot/raptor_baseline.py
results/frozen_opt/quality_gpt4o_sota/_meta/code_snapshot/run_benchmarks.py
results/frozen_opt/quality_gpt4o_sota/_meta/code_snapshot/segtree.py
results/frozen_opt/quality_gpt4o_sota/_meta/manifest.json
results/frozen_opt/quality_gpt4o_sota/quality_cells.json
results/frozen_opt/quality_gpt4o_sota/quality_ci.json
results/frozen_opt/quality_gpt4o_sota/quality_predictions.json
results/frozen_opt/quality_gpt4o_sota/quality_predictions_raw.json
results/frozen_opt/quality_gpt4o_sota/quality_results.csv
results/frozen_opt/quality_gpt4o_sota/quality_results.json
results/frozen_opt/quality_t3/_meta/code_snapshot/CHECKSUMS.txt
results/frozen_opt/quality_t3/_meta/code_snapshot/SNAPSHOT.txt
results/frozen_opt/quality_t3/_meta/code_snapshot/bm25.py
results/frozen_opt/quality_t3/_meta/code_snapshot/ce_rerank.py
results/frozen_opt/quality_t3/_meta/code_snapshot/faiss_utils.py
results/frozen_opt/quality_t3/_meta/code_snapshot/inference.py
results/frozen_opt/quality_t3/_meta/code_snapshot/navigated_fill.py
results/frozen_opt/quality_t3/_meta/code_snapshot/packing.py
results/frozen_opt/quality_t3/_meta/code_snapshot/raptor_baseline.py
results/frozen_opt/quality_t3/_meta/code_snapshot/run_benchmarks.py
results/frozen_opt/quality_t3/_meta/code_snapshot/segtree.py
results/frozen_opt/quality_t3/_meta/manifest.json
results/frozen_opt/quality_t3/quality_cells.json
results/frozen_opt/quality_t3/quality_ci.json
results/frozen_opt/quality_t3/quality_predictions.json
results/frozen_opt/quality_t3/quality_predictions_raw.json
results/frozen_opt/quality_t3/quality_results.csv
results/frozen_opt/quality_t3/quality_results.json
results/frozen_opt/summary_vs_leaf/narrativeqa.json
results/frozen_opt/summary_vs_leaf/narrativeqa_rescored.json
results/frozen_opt/summary_vs_leaf/qasper.json
results/frozen_opt/summary_vs_leaf/results.json
results/hybrid_flat_run/multihop/2wikimultihopqa/2wikimultihopqa_checkpoint.json
results/hybrid_flat_run/multihop/2wikimultihopqa/2wikimultihopqa_results.csv
results/hybrid_flat_run/multihop/2wikimultihopqa/2wikimultihopqa_results.json
results/hybrid_flat_run/multihop/2wikimultihopqa_checkpoint.json
results/hybrid_flat_run/multihop/2wikimultihopqa_results.csv
results/hybrid_flat_run/multihop/2wikimultihopqa_results.json
results/hybrid_flat_run/multihop/_pre_g_backup/2wikimultihopqa_checkpoint.json
results/hybrid_flat_run/multihop/_pre_g_backup/hotpotqa_checkpoint.json
results/hybrid_flat_run/multihop/_pre_g_backup/musique_checkpoint.json
results/hybrid_flat_run/multihop/hotpotqa/hotpotqa_checkpoint.json
results/hybrid_flat_run/multihop/hotpotqa/hotpotqa_results.csv
results/hybrid_flat_run/multihop/hotpotqa/hotpotqa_results.json
results/hybrid_flat_run/multihop/hotpotqa_checkpoint.json
results/hybrid_flat_run/multihop/hotpotqa_results.csv
results/hybrid_flat_run/multihop/hotpotqa_results.json
results/hybrid_flat_run/multihop/musique/musique_checkpoint.json
results/hybrid_flat_run/multihop/musique/musique_results.csv
results/hybrid_flat_run/multihop/musique/musique_results.json
results/hybrid_flat_run/multihop/musique_checkpoint.json
results/hybrid_flat_run/multihop/musique_results.json
results/hybrid_flat_run/multihop/narrativeqa_full/narrativeqa_full_checkpoint.json
results/hybrid_flat_run/multihop/narrativeqa_full/narrativeqa_full_results.csv
results/hybrid_flat_run/multihop/narrativeqa_full/narrativeqa_full_results.json
results/hybrid_flat_run/multihop/narrativeqa_full_checkpoint.json
results/hybrid_flat_run/multihop/narrativeqa_full_results.json
results/hybrid_flat_run/qasper/qasper_checkpoint.json
results/hybrid_flat_run/qasper/qasper_results.csv
results/hybrid_flat_run/qasper/qasper_results.json
results/hybrid_flat_run/quality/quality_checkpoint.json
results/hybrid_flat_run/quality/quality_results.csv
results/hybrid_flat_run/quality/quality_results.json
results/multihop/2wikimultihopqa/2wikimultihopqa_checkpoint.json
results/multihop/2wikimultihopqa/2wikimultihopqa_results.csv
results/multihop/2wikimultihopqa/2wikimultihopqa_results.json
results/multihop/hotpotqa/hotpotqa_checkpoint.json
results/multihop/hotpotqa/hotpotqa_results.csv
results/multihop/hotpotqa/hotpotqa_results.json
results/multihop/musique/musique_checkpoint.json
results/multihop/musique/musique_results.csv
results/multihop/musique/musique_results.json
results/multihop/narrativeqa_full/narrativeqa_full_checkpoint.json
results/multihop/narrativeqa_full/narrativeqa_full_results.csv
results/multihop/narrativeqa_full/narrativeqa_full_results.json
results/multihop_recall/2wikimultihopqa_recall.json
results/multihop_recall/hotpotqa_recall.json
results/multihop_recall/multihop_recall.json
results/multihop_recall/musique_recall.json
results/navtree_ce_qasper/qasper_checkpoint.json
results/navtree_ce_qasper/qasper_results.csv
results/navtree_ce_qasper/qasper_results.json
results/openweight_run/llama8b_local/qasper/qasper_checkpoint.json
results/openweight_run/llama8b_local/qasper/qasper_results.csv
results/openweight_run/llama8b_local/qasper/qasper_results.json
results/openweight_run/llama8b_local/quality/quality_checkpoint.json
results/openweight_run/llama8b_local/quality/quality_results.csv
results/openweight_run/llama8b_local/quality/quality_results.json
results/openweight_run/qwen7b_local/qasper/qasper_checkpoint.json
results/openweight_run/qwen7b_local/qasper/qasper_results.csv
results/openweight_run/qwen7b_local/qasper/qasper_results.json
results/openweight_run/qwen7b_local/quality/quality_checkpoint.json
results/openweight_run/qwen7b_local/quality/quality_results.csv
results/openweight_run/qwen7b_local/quality/quality_results.json
results/paired_tests/2wikimultihopqa_paired.json
results/paired_tests/hotpotqa_paired.json
results/paired_tests/musique_paired.json
results/paired_tests/qasper_paired.json
results/paired_tests/quality_paired.json
results/paired_tests/summary.json
results/paired_tests_openweight/qasper_paired.json
results/paired_tests_openweight/quality_paired.json
results/paired_tests_openweight/summary_llama8b_local.json
results/paired_tests_openweight/summary_qwen7b_local.json
results/qasper_recall/qasper_recall.csv
results/qasper_recall/qasper_recall.json
(441 additional files not shown) You must enabled JavaScript to view entire file list.
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 784 words
Related coverage

News
Quantum Computing Report · Jul 25, 2026

Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · Aug 3, 2026

Scaling Karpathy’s LLM wiki: Why your knowledge base needs a graph
Neo4j · Aug 31, 2026

Why do models task game?
AI Alignment Forum · Aug 6, 2026