analiza aiqbench
BiG-SURE introduces a novel black-box uncertainty estimation method for LLMs and VLMs by leveraging cross-temperature semantic agreement within a bipartite graph framework. This approach significantly improves the ability of models to abstain from answering in safety-critical scenarios without requiring access to model parameters or additional training data.
BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs
arXiv:2608.30646v1 Announce Type: new Abstract: Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (black-box). We propose BiG-SURE, an uncertainty estimator based on cross-temperature semantic agreement. The method samples low-temperature responses as stable semantic anchors and high-temperature responses as probes under meaning-preserving input transformations. It then constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores and defines confidence through the normalized squared spectral energy of this matrix, with uncertainty given by its complement. This bipartite graph-based Semantic Uncertainty and Reliability Estimation (SURE) score measures whether high-temperature probes remain semantically aligned with the model's stable low-temperature belief or not. We evaluate BiG-SURE on text QA, multilingual QA, and multimodal QA tasks across multiple model families. In these experiments, BiG-SURE improves average abstention AUROC over prior black-box uncertainty estimators, while remaining simple, unsupervised, and applicable to black-box model settings.
Abstract:Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (black-box). We propose BiG-SURE, an uncertainty estimator based on cross-temperature semantic agreement. The method samples low-temperature responses as stable semantic anchors and high-temperature responses as probes under meaning-preserving input transformations. It then constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores and defines confidence through the normalized squared spectral energy of this matrix, with uncertainty given by its complement. This bipartite graph-based Semantic Uncertainty and Reliability Estimation (SURE) score measures whether high-temperature probes remain semantically aligned with the model's stable low-temperature belief or not. We evaluate BiG-SURE on text QA, multilingual QA, and multimodal QA tasks across multiple model families. In these experiments, BiG-SURE improves average abstention AUROC over prior black-box uncertainty estimators, while remaining simple, unsupervised, and applicable to black-box model settings.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2608.30646 [cs.CL]
(or arXiv:2608.30646v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.30646 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 253 cuvinte
Subiecte conexe
Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs
arXiv · 20 iul. 2026
From Sequence to Structure: Relational Uncertainty Propagation for LLM Agents
arXiv · 18 aug. 2026
A preliminary experiment regarding consistency as a measure of conceptual abilities in language models
LessWrong · 17 iun. 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · 2 oct. 2026