aiqbench analysis
This research introduces attention-guided strategies—Attention-JSD, Attention-Entropy-Max, and Attention-Entropy-Min—to enhance the factuality of Large Language Models by leveraging internal self-attention mechanisms rather than relying solely on output vocabulary distributions. Experimental results indicate that these methods consistently outperform existing contrastive decoding techniques like DoLa, particularly in resolving multi-answer factual queries. By utilizing structural information from attention layers as a more sensitive signal for layer selection, this work offers a refined approach to improving LLM reliability without altering model architecture or training data.
Attention-Guided Layer Selection for Contrastive Decoding in Large Language Models
arXiv:2607.23067v1 Announce Type: cross Abstract: Contrastive decoding methods such as DoLa improve the factuality of Large Language Models (LLMs) by contrasting the output distributions of mature and premature layers. However, DoLa's dynamic layer selection relies solely on divergences in output vocabulary distributions. In this work, we propose three attention-guided strategies: Attention-JSD, Attention-Entropy-Max, and Attention-Entropy-Min, which leverage structural information carried by internal self-attention mechanisms as a signal for layer selection. Experimental results on TruthfulQA demonstrate that our strategies, particularly Attention-JSD and Attention-Entropy-Min, consistently outperform the original DoLa. We observe significant gains on multi-answer metrics (MC2 and MC3), suggesting that attention distributions can provide a more sensitive signal for resolving factual knowledge than output vocabulary distributions.
Abstract:Contrastive decoding methods such as DoLa improve the factuality of Large Language Models (LLMs) by contrasting the output distributions of mature and premature layers. However, DoLa's dynamic layer selection relies solely on divergences in output vocabulary distributions. In this work, we propose three attention-guided strategies: Attention-JSD, Attention-Entropy-Max, and Attention-Entropy-Min, which leverage structural information carried by internal self-attention mechanisms as a signal for layer selection. Experimental results on TruthfulQA demonstrate that our strategies, particularly Attention-JSD and Attention-Entropy-Min, consistently outperform the original DoLa. We observe significant gains on multi-answer metrics (MC2 and MC3), suggesting that attention distributions can provide a more sensitive signal for resolving factual knowledge than output vocabulary distributions.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2607.23067 [cs.CL]
(or arXiv:2607.23067v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2607.23067 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 207 words
Related coverage
Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning
arXiv · Sep 11, 2026
Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model
AMD · Jul 24, 2026
Model Weight Profiles: Where Do the Parameters Go?
AMD · Sep 25, 2026
Model Quantization: Post-Training Quantization Using NVIDIA Model Optimizer
NVIDIA · May 7, 2026