analiza aiqbench
This arXiv paper uses LoRA adapters on synthetic facts to show that language models stay confident when they are wrong. Output entropy tracks how sharply a model commits to an answer, not whether that answer is correct, and the gap does not close as models scale. For readers building or evaluating LLMs, the practical takeaway is that confidence-based hallucination detection needs signals tied to where an answer comes from, not just output entropy.
The Circuit Basis of Metacognitive Failure in Language Models: Interference and Hallucination
arXiv:2605.05686v4 Announce Type: replace Abstract: Language models often answer with high confidence, as measured by output entropy, even when they are wrong. We study two situations that lead to such errors, knowledge conflict, where the context contradicts a fact stored in the weights and the two sources interfere, and missing knowledge, where the fact was never learned and the model hallucinates. To control what the model knows, we fine-tune it on synthetic facts that map arbitrary entity identifiers to random codes, installed through LoRA adapters on the MLP, query-key (QK), or value-output (VO) components of the transformer. Our analysis suggests that MLP is the main substrate for storing facts in basins, regions where a fact's representations converge, while QK steers generation toward a basin or the context, and VO writes the retrieved content into the residual stream. Under knowledge conflict, the form of interference depends on how the fact is learned. A fact learned from a single phrasing yields to the context, with each adapter leaving an error pattern consistent with its circuit role. A fact learned from many phrasings overrides the context, and under contradiction the answer starts on the stored code and drifts away from it. Under missing knowledge, the hidden state reaches no basin, yet every adapter placement produces confident hallucinations, VO most of all. In both situations, output entropy reflects how sharply the model commits rather than whether the answer is correct, even though the model's representations still separate correct answers from errors. Across 12 pretrained models, the hallucination rate falls with scale, but a growing fraction of the remaining hallucinations are confident. In the models we study, the gap between confidence and correctness does not close with scale, and flagging these errors requires confidence that reflects where the answer comes from, rather than how sharply the model commits to it.
Abstract:Language models often answer with high confidence, as measured by output entropy, even when they are wrong. We study two situations that lead to such errors, knowledge conflict, where the context contradicts a fact stored in the weights and the two sources interfere, and missing knowledge, where the fact was never learned and the model hallucinates. To control what the model knows, we fine-tune it on synthetic facts that map arbitrary entity identifiers to random codes, installed through LoRA adapters on the MLP, query-key (QK), or value-output (VO) components of the transformer. Our analysis suggests that MLP is the main substrate for storing facts in basins, regions where a fact's representations converge, while QK steers generation toward a basin or the context, and VO writes the retrieved content into the residual stream. Under knowledge conflict, the form of interference depends on how the fact is learned. A fact learned from a single phrasing yields to the context, with each adapter leaving an error pattern consistent with its circuit role. A fact learned from many phrasings overrides the context, and under contradiction the answer starts on the stored code and drifts away from it. Under missing knowledge, the hidden state reaches no basin, yet every adapter placement produces confident hallucinations, VO most of all. In both situations, output entropy reflects how sharply the model commits rather than whether the answer is correct, even though the model's representations still separate correct answers from errors. Across 12 pretrained models, the hallucination rate falls with scale, but a growing fraction of the remaining hallucinations are confident. In the models we study, the gap between confidence and correctness does not close with scale, and flagging these errors requires confidence that reflects where the answer comes from, rather than how sharply the model commits to it.
Artificial Intelligence (cs.AI)
arXiv:2605.05686 [cs.AI]
(or arXiv:2605.05686v4 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2605.05686 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 392 cuvinte