analiza aiqbench
This research introduces Lowest Span Confidence (LSC), a zero-shot metric for detecting LLM hallucinations that operates efficiently on a single API response without requiring repeated sampling or internal model access. By analyzing local confidence across adjacent word spans, the method effectively captures factual uncertainty while remaining competitive with more resource-intensive approaches across major model families like Llama and Qwen. It addresses a critical reliability challenge in high-stakes AI deployment by offering a practical solution for monitoring output quality under minimal computational constraints.
Lowest Span Confidence: Zero-Shot Hallucination Detection from a Single LLM Response
arXiv:2601.19918v2 Announce Type: replace Abstract: Hallucinations in Large Language Models (LLMs), i.e., plausible but non-factual generations, pose a significant challenge to reliable deployment in high-stakes environments. However, many existing hallucination detectors require expensive repeated sampling for consistency checks or access to model-internal states unavailable in common API-based scenarios. To this end, we propose an efficient zero-shot metric called Lowest Span Confidence (LSC) for hallucination detection under minimal resource assumptions. Concretely, LSC evaluates the local confidence of adjacent complete-word spans. By selecting the lowest aggregated confidence across neighboring words whose token widths can vary, LSC captures localized uncertainty associated with factual inconsistency. This boundary-aligned smoothing reduces the global dilution of perplexity and the sensitivity of minimum token probability to isolated noise. Our main evaluation spans four model families {Llama-2, Qwen2.5, Gemma-2, Mistral} and seven benchmarks {NQ, TriviaQA, SQuAD, CoQA, HotpotQA, RAGTruth, FELM}. Additional analyses examine word reconstruction, span width, and the role of adjacency in preserving local confidence. Across these settings, LSC is competitive with methods that use multiple responses or model-internal information while requiring only one response and its output token probabilities, without training a separate detector or using an auxiliary model.
Abstract:Hallucinations in Large Language Models (LLMs), i.e., plausible but non-factual generations, pose a significant challenge to reliable deployment in high-stakes environments. However, many existing hallucination detectors require expensive repeated sampling for consistency checks or access to model-internal states unavailable in common API-based scenarios. To this end, we propose an efficient zero-shot metric called Lowest Span Confidence (LSC) for hallucination detection under minimal resource assumptions. Concretely, LSC evaluates the local confidence of adjacent complete-word spans. By selecting the lowest aggregated confidence across neighboring words whose token widths can vary, LSC captures localized uncertainty associated with factual inconsistency. This boundary-aligned smoothing reduces the global dilution of perplexity and the sensitivity of minimum token probability to isolated noise. Our main evaluation spans four model families {Llama-2, Qwen2.5, Gemma-2, Mistral} and seven benchmarks {NQ, TriviaQA, SQuAD, CoQA, HotpotQA, RAGTruth, FELM}. Additional analyses examine word reconstruction, span width, and the role of adjacency in preserving local confidence. Across these settings, LSC is competitive with methods that use multiple responses or model-internal information while requiring only one response and its output token probabilities, without training a separate detector or using an auxiliary model.
Computation and Language (cs.CL)
arXiv:2601.19918 [cs.CL]
(or arXiv:2601.19918v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2601.19918 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 280 cuvinte
Subiecte conexe
SKstars at SHROOM: Visions Agreement-Guided Ensembling of Zero-Shot and LoRA-Adapted Vision--Language Models
arXiv · 22 sept. 2026
Building trust in AI agents: A production-ready evaluation framework with Amazon Bedrock AgentCore
AWS · 16 oct. 2026
[AINews] AI Engineer NYC: Gemini 4 Argon, Vals AI Benchmarks
Latent Space · 14 oct. 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · 29 sept. 2026