aiqbench analysis
This paper introduces Agent Step Value (ASV) to audit how step rewards in black-box agents fluctuate when evaluated across different channels, revealing significant instability where cross-channel gain signs disagree nearly half the time. By isolating state-by-channel interactions in frozen PubMed QA tasks, the authors demonstrate that generated support artifacts can inflate pre-retrieval baselines, offering a method to localize these sensitivities before rewards are reused. This work provides a critical diagnostic tool for improving the robustness and reliability of agent evaluation pipelines.
Agent Step Value: Auditing Evaluation-Channel Sensitivity in Black-Box Agent Traces
arXiv:2607.04419v5 Announce Type: replace Abstract: Reusing evaluator-derived step rewards across evaluation channels raises a measurement question: does the assigned direction remain stable? We define Agent Step Value (ASV) as a channel-indexed before/after target-margin gain and identify the state-by-channel interaction from a matched 2x2 design. Across frozen PubMed question-answering transitions, cross-channel gain signs disagree on 48.3% of comparisons, compared with 0.3% for same-channel retries. Transition interactions are nearly balanced in sign (median +0.014; 10% trimmed mean -0.033), but asymmetric negative magnitudes produce a positive direct mean and a negative generated-view mean under the empirical target mixture. Equal-target weighting preserves a negative mean interaction but not this strict mean reversal. Matched retrieval designs localize the negative component to the generated-view coordinate and retain its direction across three sign checks that vary readout and evaluator stack. In a matched 512-token source-contract follow-up, source-only generation restores the positive mean direction on artifact-bearing retrievals and removes parser-detected support claims from artifact-free before-state views. These results support an explanation in which generated support inflates the pre-retrieval baseline. The resulting audit can localize channel sensitivity and test an intervention before step rewards are reused.
Abstract:Reusing evaluator-derived step rewards across evaluation channels raises a measurement question: does the assigned direction remain stable? We define Agent Step Value (ASV) as a channel-indexed before/after target-margin gain and identify the state-by-channel interaction from a matched 2x2 design. Across frozen PubMed question-answering transitions, cross-channel gain signs disagree on 48.3% of comparisons, compared with 0.3% for same-channel retries. Transition interactions are nearly balanced in sign (median +0.014; 10% trimmed mean -0.033), but asymmetric negative magnitudes produce a positive direct mean and a negative generated-view mean under the empirical target mixture. Equal-target weighting preserves a negative mean interaction but not this strict mean reversal. Matched retrieval designs localize the negative component to the generated-view coordinate and retain its direction across three sign checks that vary readout and evaluator stack. In a matched 512-token source-contract follow-up, source-only generation restores the positive mean direction on artifact-bearing retrievals and removes parser-detected support claims from artifact-free before-state views. These results support an explanation in which generated support inflates the pre-retrieval baseline. The resulting audit can localize channel sensitivity and test an intervention before step rewards are reused.
Accepted at the REALM Workshop @ EMNLP 2026 (2nd Workshop for Research on Agent Language Models). 16 pages, 5 figures. The two authors contributed equally
Artificial Intelligence (cs.AI)
arXiv:2607.04419 [cs.AI]
(or arXiv:2607.04419v5 [cs.AI] for this version)
Full text under the publisher's open licence · 218 words
Related coverage
Agent Step Value: Auditing Evaluator-Channel Reversals in Black-Box Agent Traces
arXiv · Jul 20, 2026
[AINews] not much happened today
Latent Space · Sep 18, 2026
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · Aug 3, 2026