aiqbench analysis
This study provides a robustness analysis of supervised uncertainty quantification ensembles for detecting LLM hallucinations in closed-book settings. The research demonstrates that combining heterogeneous UQ scorers significantly outperforms individual methods across various generation regimes and distribution shifts, often requiring as few as 100 labeled instances to realize gains. Furthermore, it highlights that sampling-based black-box approaches are nearly as effective as full ensembles, offering practical insights for deploying reliable detection systems without access to external tools or documents.
When Do Supervised UQ Ensembles Improve LLM Hallucination Detection? A Robustness Study
arXiv:2608.24492v1 Announce Type: cross Abstract: Uncertainty quantification (UQ) methods are widely used for hallucination detection in large language models (LLMs) in closed-book settings where ground-truth evidence is unavailable at inference time. Prior work has proposed combining UQ signals via learned ensembles, but empirical investigations into the robustness of these ensembles are limited. We study a supervised ensembling framework that trains a classifier over heterogeneous UQ-based scorer outputs on a small, domain-specific dataset of labeled LLM responses, then applies it to out-of-sample hallucination classification without retrieval, tools, or reference documents. Across four LLMs, nine datasets, and three generation regimes (short-form QA, long-form generation, and code generation), we provide a systematic robustness analysis along three axes: sample efficiency, in-domain dataset transfer, and generation regime dependence. We find that supervised ensembles outperform the best individual scorer in 30 of 32 settings, with gains realized from as few as 100 labeled instances. Ensembles retain most of their advantage in cases of in-domain transfer under distribution shift, outperforming the best non-ensemble scorer in 23 of 28 transfer settings. Sampling-based black-box ensembles are nearly as effective as full ensembles, while single-generation white-box ensembles offer limited benefit.
Title:When Do Supervised UQ Ensembles Improve LLM Hallucination Detection? A Robustness Study
Abstract:Uncertainty quantification (UQ) methods are widely used for hallucination detection in large language models (LLMs) in closed-book settings where ground-truth evidence is unavailable at inference time. Prior work has proposed combining UQ signals via learned ensembles, but empirical investigations into the robustness of these ensembles are limited. We study a supervised ensembling framework that trains a classifier over heterogeneous UQ-based scorer outputs on a small, domain-specific dataset of labeled LLM responses, then applies it to out-of-sample hallucination classification without retrieval, tools, or reference documents. Across four LLMs, nine datasets, and three generation regimes (short-form QA, long-form generation, and code generation), we provide a systematic robustness analysis along three axes: sample efficiency, in-domain dataset transfer, and generation regime dependence. We find that supervised ensembles outperform the best individual scorer in 30 of 32 settings, with gains realized from as few as 100 labeled instances. Ensembles retain most of their advantage in cases of in-domain transfer under distribution shift, outperforming the best non-ensemble scorer in 23 of 28 transfer settings. Sampling-based black-box ensembles are nearly as effective as full ensembles, while single-generation white-box ensembles offer limited benefit.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2608.24492 [cs.LG]
(or arXiv:2608.24492v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.24492 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 298 words
Related coverage
Geometry-Aware Hallucination Detection in Large Language Models
arXiv · Jun 5, 2026
A Controlled Study of Decoding-Time Truthfulness Methods on Instruction-Tuned LLMs
arXiv · Jun 11, 2026
MEND: Label-Free Detection, Localisation, and Correction of Latent Hallucination in World Models
arXiv · Oct 1, 2026
Zero-shot narrative detection in social messaging
arXiv · Sep 16, 2026