Measuring Concept Content in Text from LLM Activations: ESG Evidence from Concept Vectors and Linear Probes

Abstract:Recent work shows that Large Language Models (LLMs) encode more information in their activations than they show in their responses. We ask whether the information contained in activations could be used as a source of measurement for the presence of concepts. We compare two algorithms for extraction of concepts, i.e., Recursive Feature Machine (RFM) algorithm and linear probing against an embedding baseline, surface baselines, and the model's own response using a human-annotated Environmental, Social and Governance (ESG) dataset of financial texts. The best linear probe ranges from 0.6 to 3.3 percentage points below fine-tuned domain classifiers. The probes also outscore the model's own response in all comparisons. Furthermore, the probes outperform the RFM concept vectors in all accuracy comparisons. In a synthetic mixture test, their mean-pooled score tracks the number of concept sentences in a paragraph better than the concept-vector score. Put together, a linear probe on a frozen LLM is a cost-effective and generalizable alternative to fine-tuning a domain classifier.

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); General Economics (econ.GN)

I.2.7; I.2.6; J.4

arXiv:2608.07208 [cs.CL]

(or arXiv:2608.07208v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2608.07208 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.