aiqbench analysis
The paper tests whether "lie detector" probes, which flag outputs the model internally represents as false, can catch deception that involves no outright falsehood. Across six open-source models, probes trained on standard true-false data detected false outputs well but did much worse on misleading, technically true statements. For anyone relying on interpretability tools to monitor LLM honesty, this suggests such detectors may miss a meaningful class of deceptive behaviour, though the evidence comes from a limited set of models and hand-written examples.
Probing the Limits of the Lie Detector Approach to LLM Deception
arXiv:2603.10003v2 Announce Type: replace-cross Abstract: A strand of mechanistic approaches to deception in large language models (LLMs) relies on "lie detectors" - truth probes aimed to identify model outputs that are internally represented as false. This lie detector approach to LLM deception implicitly assumes that deception is coextensive with lying. This paper challenges that assumption by experimentally investigating whether LLMs can produce deceptive statements that are not false and whether truth probes (if naively applied as deception detectors) fail to detect such behavior. In doing so, we make four core contributions: First, we provide a dataset of manually written examples of deceptive non-falsities grounded in linguistic theory. Second, we show across six open-source LLMs of different sizes and families that some models can select these deceptive non-falsities in a forced-choice task. Third, we demonstrate that while truth probes trained on a standard true-false dataset perform well at detecting false outputs, they perform significantly worse at detecting deceptive non-falsities. We view this as substantiating the worry that naive applications of truth probes as deception detectors suffer from a critical blind spot. Fourth, we explore whether training deception probes in dialogical contexts or including misleading non-false statements in the training improves the deception detection rate.
Abstract:A strand of mechanistic approaches to deception in large language models (LLMs) relies on "lie detectors" - truth probes aimed to identify model outputs that are internally represented as false. This lie detector approach to LLM deception implicitly assumes that deception is coextensive with lying. This paper challenges that assumption by experimentally investigating whether LLMs can produce deceptive statements that are not false and whether truth probes (if naively applied as deception detectors) fail to detect such behavior. In doing so, we make four core contributions: First, we provide a dataset of manually written examples of deceptive non-falsities grounded in linguistic theory. Second, we show across six open-source LLMs of different sizes and families that some models can select these deceptive non-falsities in a forced-choice task. Third, we demonstrate that while truth probes trained on a standard true-false dataset perform well at detecting false outputs, they perform significantly worse at detecting deceptive non-falsities. We view this as substantiating the worry that naive applications of truth probes as deception detectors suffer from a critical blind spot. Fourth, we explore whether training deception probes in dialogical contexts or including misleading non-false statements in the training improves the deception detection rate.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2603.10003 [cs.CL]
(or arXiv:2603.10003v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2603.10003 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 294 words
Related coverage
Accuracy and Uncertainty in Machine Deception Detection (Research White Paper)
arXiv · Oct 15, 2026
The Truth Was Never Gone: Perfect Aliasing in Compliant-Context Truth Probes
arXiv · Sep 12, 2026
Reasoning-Token Spikes Under Prompted Untruthful Responding in Large Language Models
arXiv · Oct 8, 2026
What We Learned Trying to Catch AI Liars: An Aletheia's Quest Retrospective
EleutherAI · Aug 25, 2026