analiza aiqbench
This research identifies "weakening neurons" in large language models that, despite their scarcity, exert disproportionate influence on model behavior through negative gate activations. The study reveals a consistent architectural pattern across nine different LLMs where these neurons predominantly reside in late layers, offering new insights into the internal mechanics of transformer dynamics. Such findings are crucial for understanding how specific units shape output distributions and could inform future pruning or interpretability strategies.
Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influence
arXiv:2609.18612v1 Announce Type: cross Abstract: We analyze the learned input-output behavior of GLU-based neurons in large language models (LLMs). We propose a simple analysis method: For each neuron, we compute the cosine similarities between its input (reading) and output (writing) weight vectors. In this scheme, a strong negative cosine similarity indicates the neuron weakens the direction it detects in the residual stream, so we call this a weakening neuron. This allows us to gain a number of novel insights. First, we show that nine different LLMs have similar patterns: weakening neurons appear mostly in late layers whereas their counterparts, (conditional) strengthening neurons, are frequent in early-middle layers. Second, we find that weakening neurons display surprising behavior: even though there are few, they activate often and have a large influence on model behavior. Third, weakening neurons have a strong effect on model output when gate values are negative -- which is surprising since negative gate values are not expected to encode functionality.
Abstract:We analyze the learned input-output behavior of GLU-based neurons in large language models (LLMs). We propose a simple analysis method: For each neuron, we compute the cosine similarities between its input (reading) and output (writing) weight vectors. In this scheme, a strong negative cosine similarity indicates the neuron weakens the direction it detects in the residual stream, so we call this a weakening neuron. This allows us to gain a number of novel insights. First, we show that nine different LLMs have similar patterns: weakening neurons appear mostly in late layers whereas their counterparts, (conditional) strengthening neurons, are frequent in early-middle layers. Second, we find that weakening neurons display surprising behavior: even though there are few, they activate often and have a large influence on model behavior. Third, weakening neurons have a strong effect on model output when gate values are negative -- which is surprising since negative gate values are not expected to encode functionality.
Accepted to EMNLP 2026. Supersedes arXiv:2505.17936
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2609.18612 [cs.LG]
(or arXiv:2609.18612v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.18612 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 260 cuvinte
Subiecte conexe
Does preservation make sense before we know how to revive?
LessWrong · 15 iun. 2026
Generalization and infinite width
LessWrong · 1 aug. 2026
Challenge: Hand coding weights for efficient sequence memorisation
AI Alignment Forum · 23 iul. 2026
MSE loss does not generate superposition
LessWrong · 27 iul. 2026