analiza aiqbench
This arXiv preprint ablates neurons in LSTM and Transformer language models and compares the resulting error patterns to aphasia and neurodegenerative decline. It finds that the two architectures fail differently: LSTMs degrade gradually, while Transformers show late threshold-like collapse under MLP ablation and smoother decline under attention-head ablation. The analogies to clinical syndromes are qualitative, so readers should treat them as hypotheses for computational modeling rather than evidence about human disease.
Can Language Models Get "Aphasia" From Neuronal Ablation?
arXiv:2610.28214v1 Announce Type: new Abstract: Progressive neuronal loss is the hallmark of neurodegenerative disease, yet how loss of neurons shapes the pattern of cognitive decline remains poorly understood. Here we systematically ablate neurons (computational units) in language models spanning recurrent long short-term memory (LSTM) and Transformer architectures and apply limited fine-tuning as a simple proxy for compensatory plasticity. With whole-unit ablation, LSTMs degrade gradually with syntactic and semantic faculties declining in parallel across mild-to-moderate damage. By contrast, Transformers exhibit late threshold-like collapse under feedforward multi-layer perceptron (MLP) ablation and smoother sigmoid-like decline under attention-head ablation. Qualitatively, MLP-ablated models continue to make within-category semantic substitutions (semantic paraphasias) alongside rising non-responses, paralleling Wernicke-type aphasia, while attention-head ablation produces a trend toward loss of semantic specificity similar to the semantic variant of primary progressive aphasia (svPPA). These findings indicate that feedforward and attentional pathways have functionally distinct failure modes, suggesting that architecture-dependent failure profiles may offer new computational perspectives on aphasic syndromes and neurodegenerative diseases as an in silico model. Notably, early accuracy retention under Transformer attention-head ablation qualitatively parallels pre-symptomatic reserve before the 40--60% neuronal loss reported by Gomez-Isla et al. (1996) when clinical symptoms emerge in Alzheimer's disease, whereas the steadier LSTM decline parallels the symptomatic progression described by DeKosky and Scheff (1990) and Terry et al. (1991).
Title:Can Language Models Get "Aphasia" From Neuronal Ablation?
Abstract:Progressive neuronal loss is the hallmark of neurodegenerative disease, yet how loss of neurons shapes the pattern of cognitive decline remains poorly understood. Here we systematically ablate neurons (computational units) in language models spanning recurrent long short-term memory (LSTM) and Transformer architectures and apply limited fine-tuning as a simple proxy for compensatory plasticity. With whole-unit ablation, LSTMs degrade gradually with syntactic and semantic faculties declining in parallel across mild-to-moderate damage. By contrast, Transformers exhibit late threshold-like collapse under feedforward multi-layer perceptron (MLP) ablation and smoother sigmoid-like decline under attention-head ablation. Qualitatively, MLP-ablated models continue to make within-category semantic substitutions (semantic paraphasias) alongside rising non-responses, paralleling Wernicke-type aphasia, while attention-head ablation produces a trend toward loss of semantic specificity similar to the semantic variant of primary progressive aphasia (svPPA). These findings indicate that feedforward and attentional pathways have functionally distinct failure modes, suggesting that architecture-dependent failure profiles may offer new computational perspectives on aphasic syndromes and neurodegenerative diseases as an in silico model. Notably, early accuracy retention under Transformer attention-head ablation qualitatively parallels pre-symptomatic reserve before the 40--60% neuronal loss reported by Gomez-Isla et al. (1996) when clinical symptoms emerge in Alzheimer's disease, whereas the steadier LSTM decline parallels the symptomatic progression described by DeKosky and Scheff (1990) and Terry et al. (1991).
Computation and Language (cs.CL)
arXiv:2610.28214 [cs.CL]
(or arXiv:2610.28214v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.28214 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 315 cuvinte
Subiecte conexe
Distribution-aware Language Neuron Identification in Multilingual Large Language Models
arXiv · 11 sept. 2026
Neural FOXP2 -- Language Specific Neuron Steering for Targeted Language Improvement in LLMs
arXiv · 11 iun. 2026
Safety Kernel: Mechanistic Alignment for Cross-Lingual Defense in LLMs
arXiv · 21 oct. 2026
Is GPT 2XL already clean?
LessWrong · 25 oct. 2026