Completed Pairs Hide Capped Failures: A ReVerPi Case Study of Selective Context Projection

Abstract:Context projection replaces older tool observations with compact, addressable excerpts, reducing repeated input while potentially adding evidence-retrieval turns. We study this trade-off in ReVerPi, a Pi extension with archived observations and matched full/projected continuations. In an 86-run source-reading campaign with 641 model requests, the 15 completed pairs show identical success: 12/15 per arm. Twelve further boundary runs stop, with the runner suppressing the companion whenever the first arm fails to complete. Restoring all 27 boundary runs bounds projected-minus-full success between $-$9 and +1 tasks. One omitted, selector-chosen projected continuation successfully retrieves archive text yet exhausts twelve requests; its full counterpart answers in three. The eleven jointly correct pairs form a fully observed success stratum within this recorded frame: projection reduces aggregate logical tokens by 25%, while increasing the median pair's tokens by 29% and total suffix requests from 35 to 55. Separating fitting from evaluation changes the selector's apparent tie: outside its four fitting pairs, it incurs one extra failure and 8.6% more logical tokens over thirteen comparable runs. This methodological case study connects stopping rules, known bounded failures, unexecuted companions, and resource aggregation. Its findings concern the recorded campaign, rather than population noninferiority or superiority over unrestricted Pi. Evaluations should retain every intervention boundary, execute both allocated arms independently of the first arm's completion, and report completion alongside interaction and token expenditure.

15 pages, 12 tables, 5 figures. Project code: this https URL. Source archive includes anc/ analysis data and reproduction scripts

Artificial Intelligence (cs.AI)

arXiv:2609.31381 [cs.AI]

(or arXiv:2609.31381v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2609.31381 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

MANIFEST_SHA256.json

THIRD_PARTY_NOTICES.md

analysis/derive.py

analysis/make_assets.py

analysis/test_data.py

analysis/verify_files.py

data/PATHSPEC_LICENSE.txt

data/actual_observation.txt

data/analysis_v3.json

data/census.json

data/example_gold.json

data/pathspec_PUBLIC.json

data/pathspec_TASK.json

data/pathspec_events.json

data/pathspec_study.json

data/public_tools.json

data/quota_signal_evidence.json

data/source_inventory.csv

data/training_artifacts.json

requirements.txt

(16 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.