aiqbench analysis
This research introduces CHARM, a graph neural network framework that unifies attention maps and activation traces into attributed graphs to detect LLM hallucinations more effectively than existing heuristics. By casting detection as a graph learning task, the method demonstrates superior performance across diverse benchmarks while offering promising zero-shot transfer capabilities without requiring extensive retraining on new datasets.
Neural Message-Passing on Attention Graphs for Hallucination Detection
arXiv:2509.24770v2 Announce Type: replace Abstract: Large Language Models (LLMs) often generate incorrect or unsupported content, known as hallucinations. Existing detection methods rely on heuristics or simple models over isolated computational traces such as activations, or attention maps. We unify these signals by representing them as attributed graphs, where tokens are nodes, edges follow attentional flows, and both carry features from attention scores and activations. Our approach, CHARM, casts hallucination detection as a graph learning task and tackles it by applying GNNs over the above attributed graphs. We show that CHARM provably subsumes prior attention-based heuristics and, experimentally, it consistently outperforms other leading approaches across diverse benchmarks. Our results shed light on the relevant role played by the graph structure and on the benefits of combining computational traces, whilst showing CHARM exhibits promising zero-shot performance on cross-dataset transfer.
Abstract:Large Language Models (LLMs) often generate incorrect or unsupported content, known as hallucinations. Existing detection methods rely on heuristics or simple models over isolated computational traces such as activations, or attention maps. We unify these signals by representing them as attributed graphs, where tokens are nodes, edges follow attentional flows, and both carry features from attention scores and activations. Our approach, CHARM, casts hallucination detection as a graph learning task and tackles it by applying GNNs over the above attributed graphs. We show that CHARM provably subsumes prior attention-based heuristics and, experimentally, it consistently outperforms other leading approaches across diverse benchmarks. Our results shed light on the relevant role played by the graph structure and on the benefits of combining computational traces, whilst showing CHARM exhibits promising zero-shot performance on cross-dataset transfer.
Machine Learning (cs.LG)
arXiv:2509.24770 [cs.LG]
(or arXiv:2509.24770v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2509.24770 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 224 words
Related coverage
Temporal Generalization and Explanation Stability of Control Flow Graph Neural Networks for Malware Detection
arXiv · Sep 22, 2026
Diffusion-Induced Spatial Attention Overlapping Community Detection
arXiv · Sep 23, 2026
A Weisfeiler-Leman Characterization of Global-Attention Graph Transformers for Mixed-Integer Linear Programs
arXiv · Jul 21, 2026
An Attention-Driven Heterogeneous GNN Model for Credit Card Fraud Detection
arXiv · Sep 30, 2026