analiza aiqbench
This research proposes a method to override the English dominance in large language models by identifying and steering sparse, low-rank language-specific neurons. By isolating these control circuits, the technique allows for targeted activation shifts that prioritize non-English languages like Hindi or Spanish without retraining the entire model. The approach offers a mechanistic pathway to address parametric memory suppression, potentially enabling more equitable multilingual capabilities through precise internal intervention.
Neural FOXP2 -- Language Specific Neuron Steering for Targeted Language Improvement in LLMs
arXiv:2602.00945v2 Announce Type: replace Abstract: LLMs are multilingual by training, yet their lingua franca is often English, reflecting English language dominance in pretraining. Other languages remain in parametric memory but are systematically suppressed. We argue that language defaultness is governed by a sparse, low-rank control circuit, language neurons, that can be mechanistically isolated and safely steered. We introduce Neural FOXP2, that makes a chosen language (Hindi or Spanish) primary in a model by steering language-specific neurons. Neural FOXP2 proceeds in three stages: (i) Localize: We train per-layer SAEs so each activation decomposes into a small set of active feature components. For every feature, we quantify English vs. Hindi/Spanish selectivity overall logit-mass lift toward the target-language token set. Tracing the top-ranked features back to their strongest contributing units yields a compact language-neuron set. (ii) Steering directions: We localize controllable language-shift geometry via a spectral low-rank analysis. For each layer, we build English to target activation-difference matrices and perform layerwise SVD to extract the dominant singular directions governing language change. The eigengap and effective-rank spectra identify a compact steering subspace and an empirically chosen intervention window (where these directions are strongest and most stable). (iii) Steer: We apply a signed, sparse activation shift targeted to the language neurons. Concretely, within low to mid layers we add a positive steering along the target-language dominant directions and a compensating negative shift toward the null space for the English neurons, yielding controllable target-language defaultness.
Abstract:LLMs are multilingual by training, yet their lingua franca is often English, reflecting English language dominance in pretraining. Other languages remain in parametric memory but are systematically suppressed. We argue that language defaultness is governed by a sparse, low-rank control circuit, language neurons, that can be mechanistically isolated and safely steered. We introduce Neural FOXP2, that makes a chosen language (Hindi or Spanish) primary in a model by steering language-specific neurons. Neural FOXP2 proceeds in three stages: (i) Localize: We train per-layer SAEs so each activation decomposes into a small set of active feature components. For every feature, we quantify English vs. Hindi/Spanish selectivity overall logit-mass lift toward the target-language token set. Tracing the top-ranked features back to their strongest contributing units yields a compact language-neuron set. (ii) Steering directions: We localize controllable language-shift geometry via a spectral low-rank analysis. For each layer, we build English to target activation-difference matrices and perform layerwise SVD to extract the dominant singular directions governing language change. The eigengap and effective-rank spectra identify a compact steering subspace and an empirically chosen intervention window (where these directions are strongest and most stable). (iii) Steer: We apply a signed, sparse activation shift targeted to the language neurons. Concretely, within low to mid layers we add a positive steering along the target-language dominant directions and a compensating negative shift toward the null space for the English neurons, yielding controllable target-language defaultness.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2602.00945 [cs.CL]
(or arXiv:2602.00945v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2602.00945 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 330 cuvinte
Subiecte conexe
Beyond Accuracy: What Language-Subspace Interventions Capture, Remove, and Improve
arXiv · 4 nov. 2026
Scaling Multilingual Pretraining of Pixel Language Models
arXiv · 22 oct. 2026
Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering
arXiv · 19 oct. 2026
Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering
arXiv · 12 iun. 2026