aiqbench analysis
This research investigates how deictic shifts—where context-dependent terms like "previous" refer to different entities across pipeline stages—affect accuracy in draft-verify-revise LLM orchestration patterns. The study demonstrates that while increased reasoning effort can significantly improve performance, meta-evaluators often rely on surface cues rather than operational logic when errors occur, suggesting a need for explicit referent management by context engineers.
Can LLMs in Draft-Verify-Revise Pipelines Resolve Deictic Ambiguity?
arXiv:2609.12162v1 Announce Type: cross Abstract: Draft-verify-revise is a common LLM orchestration pattern for scaling inference-time compute. One LLM drafts, a second critiques the draft and provides feedback, and a third uses that feedback to revise the draft into the final output. As context cascades between stages, LLMs at different stages can resolve a context-dependent expression such as "previous" differently. When that happens, the expression undergoes a deictic shift, a change in what it refers to. This phenomenon was studied with a synthetic dataset of 10 base examples, each rendered in three conditions. Holding the shared components constant, the conditions varied whether the draft stage LLM (the assistant) or the verify stage LLM (the grader) resolved the expression correctly, and how much independent reasoning the revise stage LLM (the meta-evaluator) needed to determine which reading was correct. Six models from three providers were tested across 21 reasoning effort configurations using e-values for sequential testing, in a primary experiment and an ablation experiment that removed error classification labels from the grader's feedback. A separate LLM analyzed the meta-evaluator's stated rationale for each wrong verdict. Balanced accuracy (the unweighted mean of sensitivity and specificity) ranged from 0.156, below chance, to near-perfect. GPT-5.2 rose from 0.156 without reasoning to 0.942 at its highest reasoning effort level, while Gemini 3 Pro stayed above 0.94 at every level. Gemini 3 Pro at low reasoning effort outscored GPT-5.2 at xhigh reasoning effort for roughly 5% of the cost per trial. When the meta-evaluator erred, it tended to rely on surface cues rather than operational reasoning. Context engineers implementing draft-verify-revise pipelines should be wary of deictic shifts and make the intended referent explicit at each stage.
Abstract:Draft-verify-revise is a common LLM orchestration pattern for scaling inference-time compute. One LLM drafts, a second critiques the draft and provides feedback, and a third uses that feedback to revise the draft into the final output. As context cascades between stages, LLMs at different stages can resolve a context-dependent expression such as "previous" differently. When that happens, the expression undergoes a deictic shift, a change in what it refers to. This phenomenon was studied with a synthetic dataset of 10 base examples, each rendered in three conditions. Holding the shared components constant, the conditions varied whether the draft stage LLM (the assistant) or the verify stage LLM (the grader) resolved the expression correctly, and how much independent reasoning the revise stage LLM (the meta-evaluator) needed to determine which reading was correct. Six models from three providers were tested across 21 reasoning effort configurations using e-values for sequential testing, in a primary experiment and an ablation experiment that removed error classification labels from the grader's feedback. A separate LLM analyzed the meta-evaluator's stated rationale for each wrong verdict. Balanced accuracy (the unweighted mean of sensitivity and specificity) ranged from 0.156, below chance, to near-perfect. GPT-5.2 rose from 0.156 without reasoning to 0.942 at its highest reasoning effort level, while Gemini 3 Pro stayed above 0.94 at every level. Gemini 3 Pro at low reasoning effort outscored GPT-5.2 at xhigh reasoning effort for roughly 5% of the cost per trial. When the meta-evaluator erred, it tended to rely on surface cues rather than operational reasoning. Context engineers implementing draft-verify-revise pipelines should be wary of deictic shifts and make the intended referent explicit at each stage.
86 pages including supplementary materials, 4 figures, 10 supplementary figures, 17 supplementary tables, 11 supplementary exemplars. Code and data: this https URL
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2609.12162 [cs.AI]
(or arXiv:2609.12162v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.12162 arXiv-issued DOI via DataCite (pending registration)
From: Obinna Ekekezie M.D. [view email] [v1] Thu, 10 Sep 2026 19:49:06 UTC (16,058 KB)
Full text under the publisher's open licence · 332 words
Related coverage
Automated Reasoning policy refinement in Amazon Bedrock
AWS · Aug 3, 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026
A Red Line and Oversight Framework for Government AI Contracts
AI Alignment Forum · Jul 18, 2026
An Organizational Second Brain: Building an AI That Learns From Experts
Meta Engineering · Sep 2, 2026