analiza aiqbench
This research introduces Kernel Token Contradiction (KTC), a novel method for quantifying claim-level uncertainty in Large Language Models by integrating conditional distributions with token contradiction scores derived from Wikipedia statistics. By leveraging Von Neumann entropy on a positive semi-definite kernel, the approach achieves significant speedups over existing GPU-accelerated and CPU-based methods while maintaining high accuracy across multiple languages and models. The technique enables practical, real-time monitoring of LLM reliability in production environments without requiring specialized hardware.
Kernel Token Contradiction: a Fast and Principled Approach for LLM Claim Uncertainty Quantification
arXiv:2608.22506v1 Announce Type: new Abstract: Claim-level Uncertainty Quantification (UQ) aims to mitigate the lack of reliability of Large Language Models (LLMs) by evaluating the factuality of each claim in their outputs. We introduce Kernel Token Contradiction (KTC), a lightweight approach to compute claim-level UQ under realistic white-box conditions. KTC represents the candidate tokens involved in LLM generation as a positive semi-definite kernel that integrates both the LLM's conditional distribution and a token contradiction score. We then use the Von Neumann entropy to quantify the uncertainty of this kernel. To estimate token contradiction, we develop a new approach based on frequency statistics from the Wikipedia corpus. Although CPU-only, our approach achieves over an 8.2x speedup compared to state-of-the-art GPU-accelerated methods based on cross-encoders, and over a 65x speedup compared to CPU-only methods with comparable performance. Our evaluation spans two benchmarks across four European languages and 16 different models. KTC not only matches the average performance of existing methods but also outperforms them in high-precision regimes. This combination of computational efficiency and accuracy makes real-time monitoring of LLM outputs practical in production.
Abstract:Claim-level Uncertainty Quantification (UQ) aims to mitigate the lack of reliability of Large Language Models (LLMs) by evaluating the factuality of each claim in their outputs. We introduce Kernel Token Contradiction (KTC), a lightweight approach to compute claim-level UQ under realistic white-box conditions. KTC represents the candidate tokens involved in LLM generation as a positive semi-definite kernel that integrates both the LLM's conditional distribution and a token contradiction score. We then use the Von Neumann entropy to quantify the uncertainty of this kernel. To estimate token contradiction, we develop a new approach based on frequency statistics from the Wikipedia corpus. Although CPU-only, our approach achieves over an 8.2x speedup compared to state-of-the-art GPU-accelerated methods based on cross-encoders, and over a 65x speedup compared to CPU-only methods with comparable performance. Our evaluation spans two benchmarks across four European languages and 16 different models. KTC not only matches the average performance of existing methods but also outperforms them in high-precision regimes. This combination of computational efficiency and accuracy makes real-time monitoring of LLM outputs practical in production.
Preprint. Under review
Computation and Language (cs.CL)
arXiv:2608.22506 [cs.CL]
(or arXiv:2608.22506v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.22506 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 273 cuvinte
Subiecte conexe
DI Series: Serving Kimi-K3 MXFP4 Wide-EP Disaggregated on AMD Instinct MI300X and MI325X
AMD · 8 oct. 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · 13 oct. 2026
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · 16 oct. 2026
[AINews] Agents disprove Polysemanticity; greatly improve Interpretability & Alignment
Latent Space · 16 oct. 2026