aiqbench analysis
This research introduces "Perturbation," a novel method for analyzing language models by treating representations as conduits for learning rather than static activation patterns. By fine-tuning on adversarial examples, the technique effectively traces structured linguistic generalization without relying on geometric assumptions or finding spurious representations in untrained models. This approach offers a more faithful tool for understanding how large models acquire and transfer abstract linguistic concepts from experience alone.
Perturbation: A simple and efficient adversarial tracer for representation learning in language models
arXiv:2603.23821v2 Announce Type: replace-cross Abstract: Linguistic representation learning in deep neural language models (LMs) has been studied for decades, but finding representations in LMs remains an unsolved problem. On the one hand, unconstrained alignments may trivialize the notion of representation (Sutter et al., 2025); on the other, even recently popularized linear approaches may not always be faithful to natural model behavior (Arora et al. 2024). Here we escape this dilemma by reconceptualizing representations not as patterns of activation but as conduits for learning. Our approach is simple: we perturb an LM by fine-tuning it on a single adversarial example and measure how this perturbation "infects" other examples. Perturbation makes no geometric assumptions, and unlike other methods, it does not find representations where it should not (e.g., in untrained LMs). But in trained LMs, perturbation reveals structured transfer at multiple linguistic grain sizes, suggesting that LMs both generalize along representational lines and acquire linguistic abstractions from experience alone.
Abstract:Linguistic representation learning in deep neural language models (LMs) has been studied for decades, but finding representations in LMs remains an unsolved problem. On the one hand, unconstrained alignments may trivialize the notion of representation (Sutter et al., 2025); on the other, even recently popularized linear approaches may not always be faithful to natural model behavior (Arora et al. 2024). Here we escape this dilemma by reconceptualizing representations not as patterns of activation but as conduits for learning. Our approach is simple: we perturb an LM by fine-tuning it on a single adversarial example and measure how this perturbation "infects" other examples. Perturbation makes no geometric assumptions, and unlike other methods, it does not find representations where it should not (e.g., in untrained LMs). But in trained LMs, perturbation reveals structured transfer at multiple linguistic grain sizes, suggesting that LMs both generalize along representational lines and acquire linguistic abstractions from experience alone.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2603.23821 [cs.CL]
(or arXiv:2603.23821v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2603.23821 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 251 words
Related coverage
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · Sep 23, 2026
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
ICA Lens: Interpreting Language Models Without Training Another Dictionary
arXiv · Jun 12, 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · May 19, 2026