analiza aiqbench
This arXiv paper tests whether natural language autoencoders, a method for translating LLM internal activations into text, hold up beyond their original training setting. Results are mixed: verbalizations of composed concepts mostly name the target, the method transfers to lightly finetuned models (with degradation on the largest model), and it can recover hidden secret words and finetuning objectives. It still trails state-of-the-art model diffing methods on emergent misalignment detection, though it beats a blackbox baseline. Readers interested in interpretability should note that the evidence is promising but limited to four models from three families.
Verbalizing Activation Differences with Natural Language Autoencoders
arXiv:2610.30934v1 Announce Type: new Abstract: Activation verbalizers translate the internal representations of large language models into human-readable text. The recently proposed natural language autoencoders (NLAs) learn this by reconstructing activations through a natural language bottleneck. However, it remains unclear how well these methods generalize beyond the setting they were trained for. Here, we study this question for NLAs along four closely related questions: (i) Do NLA verbalizations exhibit compositional structure with respect to their input? (ii) Do NLAs generalize to finetuned versions of the model they were trained for? (iii) Can NLAs recover finetuning objectives in model diffing? (iv) Can NLAs be specialized for model diffing? Across four models from three families, our experiments provide evidence supporting a largely positive answer to all four questions. Specifically, verbalizations of composed analogy activations mostly name the target while rarely mentioning the subtracted word, and an NLA can be reused on lightweight finetunes of its original model without retraining, with reconstruction quality degrading noticeably only for the largest model. Verbalizing activation differences between Taboo model organisms and their original models recovers the concealed secret word for most organisms, and reinforcement learning on activation differences further improves this recovery on held-out organisms. We also find that NLAs can recover the finetuning objective without the finetuned model generating any token, since the most informative token positions precede the user prompt. NLAs do not yet match state-of-the-art model diffing methods on the emergent misalignment detection task, but outperform a blackbox baseline. Our findings suggest that NLAs can meaningfully verbalize differences in activation space, rendering NLAs a promising direction for future work.
Abstract:Activation verbalizers translate the internal representations of large language models into human-readable text. The recently proposed natural language autoencoders (NLAs) learn this by reconstructing activations through a natural language bottleneck. However, it remains unclear how well these methods generalize beyond the setting they were trained for. Here, we study this question for NLAs along four closely related questions: (i) Do NLA verbalizations exhibit compositional structure with respect to their input? (ii) Do NLAs generalize to finetuned versions of the model they were trained for? (iii) Can NLAs recover finetuning objectives in model diffing? (iv) Can NLAs be specialized for model diffing? Across four models from three families, our experiments provide evidence supporting a largely positive answer to all four questions. Specifically, verbalizations of composed analogy activations mostly name the target while rarely mentioning the subtracted word, and an NLA can be reused on lightweight finetunes of its original model without retraining, with reconstruction quality degrading noticeably only for the largest model. Verbalizing activation differences between Taboo model organisms and their original models recovers the concealed secret word for most organisms, and reinforcement learning on activation differences further improves this recovery on held-out organisms. We also find that NLAs can recover the finetuning objective without the finetuned model generating any token, since the most informative token positions precede the user prompt. NLAs do not yet match state-of-the-art model diffing methods on the emergent misalignment detection task, but outperform a blackbox baseline. Our findings suggest that NLAs can meaningfully verbalize differences in activation space, rendering NLAs a promising direction for future work.
Computation and Language (cs.CL)
arXiv:2610.30934 [cs.CL]
(or arXiv:2610.30934v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.30934 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 355 cuvinte
Subiecte conexe
Where Do Apparent LLM Clinical Triage Failures Arise? Localizing the Multiple-Choice Format Effect
arXiv · 6 oct. 2026
Probing Latent Colombian Identity Inferences in Qwen2.5-7B with Natural Language Autoencoders
arXiv · 28 iul. 2026
SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization
arXiv · 19 aug. 2026
Faithful Activation Verbalization: Reducing Hallucinations in LLM Representation Interpretation
arXiv · 30 sept. 2026