aiqbench analysis
ProbPlug addresses the critical gap in LLM reliability by offering a lightweight method to estimate prediction confidence without altering the base model architecture. This approach is particularly valuable for high-stakes applications where understanding uncertainty in binary classification tasks is essential before deployment. The framework demonstrates strong generalization across both text-based and multimodal models with negligible computational overhead.
ProbPlug: A Plugin Uncertainty Network for Reliable Confidence in LLM Binary Classification
arXiv:2609.10122v1 Announce Type: new Abstract: Large language models (LLMs) have achieved strong performance across a broad range of classification settings, yet the reliability of their predictions remains a major obstacle to deployment in high-stakes scenarios. Although confidence estimation for LLMs has been widely studied, confidence calibration for LLM-based classification remains underexplored. We introduce ProbPlug, a lightweight confidence estimation framework for LLM-based binary classification, which predicts whether an output is correct using internal token features extracted from a frozen LLM. ProbPlug employs a self-attention module to aggregate hidden representations and can be integrated into the original inference pipeline without modifying the base model. Experiments across multiple tasks involving both text-based and multimodal large models show that ProbPlug provides more reliable confidence estimates, improves classification performance with negligible additional overhead, and exhibits strong generalization across tasks. These results indicate that ProbPlug serves as a practical solution for confidence estimation in LLM-based classification. Our code is publicly available at Github.
Abstract:Large language models (LLMs) have achieved strong performance across a broad range of classification settings, yet the reliability of their predictions remains a major obstacle to deployment in high-stakes scenarios. Although confidence estimation for LLMs has been widely studied, confidence calibration for LLM-based classification remains underexplored. We introduce ProbPlug, a lightweight confidence estimation framework for LLM-based binary classification, which predicts whether an output is correct using internal token features extracted from a frozen LLM. ProbPlug employs a self-attention module to aggregate hidden representations and can be integrated into the original inference pipeline without modifying the base model. Experiments across multiple tasks involving both text-based and multimodal large models show that ProbPlug provides more reliable confidence estimates, improves classification performance with negligible additional overhead, and exhibits strong generalization across tasks. These results indicate that ProbPlug serves as a practical solution for confidence estimation in LLM-based classification. Our code is publicly available at Github.
Accepted by the 23rd Pacific Rim International Conference on Artificial Intelligence. (PRICAI 2026)
Computation and Language (cs.CL)
arXiv:2609.10122 [cs.CL]
(or arXiv:2609.10122v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.10122 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 260 words
Related coverage
When LLM judges agree, should we believe them?
Amazon Science · Aug 26, 2026
ProteoKnight: Convolution-based Phage Virion Protein Classification and Uncertainty Analysis
arXiv · Aug 19, 2026
[AINews] not much happened today
Latent Space · Oct 3, 2026
Uncertainty quantification for trustworthy deep learning: Methods and measures
arXiv · Jul 31, 2026