aiqbench analysis
This paper is squarely about LLM agent behavior, so it passes the topic gate and falls under AI Research rather than a product or policy category. Its main contribution is methodological: it separates a model's ability to discriminate correction from manipulation from its general willingness to concede, using signal detection theory across seven models. The agentic findings matter for anyone deploying agents with policy constraints, since answer-level concession did not predict policy violations, and benchmark concession rates alone can be misleading.
Correction or Manipulation? Measuring Epistemic Vigilance in LLM Agents Under User Pressure
arXiv:2610.36453v1 Announce Type: new Abstract: LLM agents that answer questions and act on behalf of users must decide when to yield to a user who disagrees. Disagreement may correct a genuine error or may be an attempt at manipulation. Prior work shows that models frequently abandon correct answers when challenged by users, yet it remains unclear whether they distinguish correction from manipulation or merely differ in their propensity to concede. We examine this question across seven LLMs from four vendors in two studies, one on the answers agents give and one on the actions they take as customer-support agents whose policy permits account changes only for verified users. Simulated users push toward correct and incorrect answers with graded pressure. Following research on epistemic vigilance, the pressure either appeals to the source, such as claimed authority, or offers content the agent cannot verify, namely a cited source. Drawing on signal detection theory, we measure on a set of benchmark questions how much more readily each agent concedes to correction than to manipulation, separately from its overall propensity to concede. Citations made five of seven models readier to concede and four worse at discriminating, while stronger wording added little. In the agentic study, five models never violated the verification policy, including models that readily abandoned correct answers, so concession on answers did not predict violations of policy. Ministral 3B violated the policy mainly under claimed authority, and Ministral 8B reported changes it had not made. An instruction to revise only on verifiable evidence chiefly shifted propensity to concede rather than improving discrimination. Concession rates alone are therefore misleading, and agents should be evaluated on both their discrimination and the actions they take.
Title:Correction or Manipulation? Measuring Epistemic Vigilance in LLM Agents Under User Pressure
Abstract:LLM agents that answer questions and act on behalf of users must decide when to yield to a user who disagrees. Disagreement may correct a genuine error or may be an attempt at manipulation. Prior work shows that models frequently abandon correct answers when challenged by users, yet it remains unclear whether they distinguish correction from manipulation or merely differ in their propensity to concede. We examine this question across seven LLMs from four vendors in two studies, one on the answers agents give and one on the actions they take as customer-support agents whose policy permits account changes only for verified users. Simulated users push toward correct and incorrect answers with graded pressure. Following research on epistemic vigilance, the pressure either appeals to the source, such as claimed authority, or offers content the agent cannot verify, namely a cited source. Drawing on signal detection theory, we measure on a set of benchmark questions how much more readily each agent concedes to correction than to manipulation, separately from its overall propensity to concede. Citations made five of seven models readier to concede and four worse at discriminating, while stronger wording added little. In the agentic study, five models never violated the verification policy, including models that readily abandoned correct answers, so concession on answers did not predict violations of policy. Ministral 3B violated the policy mainly under claimed authority, and Ministral 8B reported changes it had not made. An instruction to revise only on verifiable evidence chiefly shifted propensity to concede rather than improving discrimination. Concession rates alone are therefore misleading, and agents should be evaluated on both their discrimination and the actions they take.
20 pages, 7 figures, 11 tables. 8-page main paper with supplementary material included as an appendix. Under review
Artificial Intelligence (cs.AI)
I.2.7; I.2.11; I.2.0
arXiv:2610.36453 [cs.AI]
(or arXiv:2610.36453v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.36453 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 402 words
Related coverage
GAINS: Leveraging Inconsistent Human Intervention Signals in Reinforcement Learning
arXiv · Aug 18, 2026
FWBC-VLA: Force-Aware Whole-Body Compensation for Contact-Rich Loco-Manipulation
arXiv · Sep 5, 2026
AnchorRefine: Synergy-Manipulation Based on Trajectory Anchor and Residual Refinement for Vision-Language-Action Models
arXiv · Jul 23, 2026
Beyond Dexterity: Why Contact May Define the Next Era of Robotics
IEEE Spectrum · Jun 9, 2026